One way to solve it iteratively is to do an in-order traversal of the tree using explicit stack.
Then, as you traverses the tree, compare the previous node's value with the current node's value to verify all nodes' values follow a monotonically increasing order.
Better yet, you can design a BST iterator which yields a cleaner solution.
bool isBST(TreeNode *root) {
BSTIterator iterator(root);
int prev = INT_MIN;
while (iterator.hasNext()) {
int curr = iterator.next()->val;
if (prev > curr) {
return false;
}
prev = curr;
}
return true;
}