libtracecmd: Have the rbtree check be more elaborate

Update the check_tree() that verifies the rbtree (when enabled) to be a
little more elaborate to find more errors with the tree.

Link: https://lore.kernel.org/20260529143411.7ace2f59@fedora
Signed-off-by: Steven Rostedt <rostedt@goodmis.org>