| Debugging Jato |
| |
| This document is intended to help you to debug and fix bugs in Jato. You are |
| expected to have basic understanding of how the Java virtual machine works. You |
| are also expected to have working knowledge of C and Java. |
| |
| |
| Part 1: Preliminaries |
| |
| To make your life easier, you should do the following steps before you start |
| debugging: |
| |
| a) Install valgrind |
| |
| b) Install JamVM or GNU Classpath version of CACAO |
| |
| c) Download GNU Classpath sources for the version you have installed on your |
| system. |
| |
| Jato relies on GNU Classpath to provide core Java runtime libraries (e.g. |
| java.lang package) so it essential to consider GNU Classpath as part of the |
| system when debugging a bug. JamVM or CACAO is useful to have around to verify |
| that a particular bug only exists in Jato and that its not a generic problem |
| with GNU Classpath. |
| |
| |
| Part 2: Minimal test case |
| |
| The first step to debugging a problem is to create a minimal test case that |
| triggers the bug. If you're seeing a native backtrace or a Java stack trace, |
| it's usually easy to work out where the problem happened and use that |
| information to create a test case. You can also use the "-Xtrace:invoke" |
| command line option to see what methods were invoked before the crash. |
| |
| |
| Part 3: Finding the root cause |
| |
| Once you have a minimal test case that triggers the problem, you should first |
| run the test case with valgrind: |
| |
| valgrind jato -Xnogc <test case> |
| |
| The "-Xnogc" disables Boehm GC which triggers valgrind errors. If valgrind |
| reports problems when the GC is disabled, they're almost certainly related to |
| your bug. |
| |
| If your crash happens in native code: |
| |
| [main] SIGSEGV at EIP b775212b while accessing memory address 00000000. |
| [main] Registers: |
| [main] eax: 00000000 ebx: b775bff4 ecx: bf988804 edx: b775bff4 |
| [main] esi: 00000000 edi: 080a3d7c ebp: bf98886c esp: bf988804 |
| [main] Native and Java stack trace: |
| [main] [<b775212b>] native : exception_check+af6e9e3a (arch/x86/unwind_32.S:75) |
| [main] [<a74ed214>] jit : gnu/classpath/Pointer.<init>(Pointer.java) |
| [main] [<a74ebe2e>] jit : gnu/java/net/PlainSocketImpl.bind(PlainSocketImpl.java:307) |
| [main] [<a74eb885>] jit : java/net/ServerSocket.bind(ServerSocket.java:250) |
| [main] [<a74c47f6>] jit : java/net/ServerSocket.<init>(ServerSocket.java:181) |
| [main] [<a74c4539>] jit : java/net/ServerSocket.<init>(ServerSocket.java:155) |
| [main] [<a74c44f5>] jit : java/net/ServerSocket.<init>(ServerSocket.java:137) |
| [main] [<a74c44a5>] jit : ServerSocketTest.main(ServerSocketTest.java:6) |
| [main] [<0807bc65>] native : do_main_class+141 (/home/penberg/src/jato/vm/jato.c:1036) |
| [main] [<0807c1f7>] native : ./jato(main+0x582) [0x807c1f7] |
| [main] [<b7442bd5>] native : exception_check+af3da8e4 (arch/x86/unwind_32.S:75) |
| [main] [<0805cb40>] native : ./jato() [0x805cb40] |
| Aborted |
| |
| You can use the "addr2line" command to display better information on where the |
| crash happens: |
| |
| addr2line -e jato |
| <EIP> |
| |
| If the crash happens in JIT code: |
| |
| [main] SIGSEGV at RIP 00000000 while accessing memory address 00000000. |
| [main] Registers: |
| [main] rsp: 00007ffffab5e848 |
| [main] rax: 0000000001eaf240 rbx: 0000000002778e70 rcx: 0000000000000017 |
| [main] rdx: 0000000002767540 rsi: 000000000277adc0 rdi: 000000000277adc0 |
| [main] rbp: 00007ffffab5eb00 r8: 000000000000000b r9: 00007ffffab5eb00 |
| [main] r10: 3b70614d2f6c6974 r11: 0000000000000246 r12: 00007f40c73c86f0 |
| [main] r13: 0000000002767540 r14: 0000000001d33e38 r15: 000000000277ab40 |
| [main] Native and Java stack trace: |
| [main] [<00000000>] native : [(nil)] |
| [main] [<41b547c7>] jit : java/util/Hashtable.clone(Hashtable.java:558) |
| [main] [<41b4f386>] jit : gnu/classpath/SystemProperties.<clinit>(SystemProperties.java:125) |
| [main] [<00435677>] native : vm_class_init+131 (/home/penberg/jato/vm/class.c:648) |
| [main] [<0042e8e8>] native : vm_class_ensure_init+10c68 (/home/penberg/jato/include/vm/class.h:97) |
| [main] [<41b3a5b2>] jit : gnu/classpath/SystemProperties.getProperty(SystemProperties.java) |
| [main] [<41b4de2e>] jit : java/lang/VMClassLoader.getResources(VMClassLoader.java:184) |
| [main] [<41b49294>] jit : java/lang/VMClassLoader.getResource(VMClassLoader.java:170) |
| [main] [<41b48b35>] jit : java/lang/VMClassLoader.getBootPackages(VMClassLoader.java:252) |
| [main] [<41b3eed5>] jit : java/lang/VMClassLoader.<clinit>(VMClassLoader.java:89) |
| [main] [<00435677>] native : vm_class_init+131 (/home/penberg/jato/vm/class.c:648) |
| [main] [<0042e8e8>] native : vm_class_ensure_init+10c68 (/home/penberg/jato/include/vm/class.h:97) |
| [main] [<41b3a0d2>] jit : java/lang/VMClassLoader.getSystemClassLoader(VMClassLoader.java) |
| [main] [<41b3c23b>] jit : java/lang/ClassLoader$StaticData.<clinit>(ClassLoader.java:154) |
| [main] [<00435677>] native : vm_class_init+131 (/home/penberg/jato/vm/class.c:648) |
| [main] [<0041b95e>] native : vm_class_ensure_init+ffffffffffffdcde (/home/penberg/jato/include/vm/class.h:97) |
| [main] [<00443326>] native : static_field_signal_bh+e (/home/penberg/jato/vm/static.c:66) |
| |
| You can use the "-Xtrace:jit" command line option to trace the generated code: |
| |
| jato -Xtrace:jit <test case> &> trace |
| |
| and look up machine code where EIP resides in. |
| |
| |
| Part 4: Debugging deadlocks |
| |
| If a program seems to deadlock under Jato, you can use GDB to capture a |
| backtrace of all the threads while the program is stuck: |
| |
| gdb --args ./jato [...] |
| |
| (gdb) handle SIGSEGV nostop |
| (gdb) r |
| <Ctrl-C when the program deadlocks> |
| (gdb) thread apply all backtrace |
| |
| |
| Part 5: Using GDB with JITed code (only x86-64 currently) |
| |
| Jato offers special GDB support. Otherwise you wouldn't be able to break into |
| JITed code and get meaningful backtraces. However, you'll need to rebuild |
| Jato like this: |
| |
| make clean |
| make DEBUG=1 |
| |
| Let's see what we can do now: |
| |
| gdb --args ./jato [...] |
| (gdb) handle SIGSEGV nostop |
| (gdb) break java_io_File_checkRead |
| Function "java_io_File_checkRead" not defined. |
| Make breakpoint pending on future shared library load? (y or [n]) y |
| Breakpoint 1 (java_io_File_checkRead) pending. |
| (gdb) r |
| [...] |
| (gdb) bt |
| #0 0x00000000400fe374 in java_io_File_checkRead () |
| #1 0x00000000400fe22e in java_io_File_isDirectory () |
| #2 0x00000000400e7f7d in java_lang_VMClassLoader_getResources () |
| #3 0x00000000400e338b in java_lang_VMClassLoader_getResource () |
| #4 0x00000000400e2cdc in java_lang_VMClassLoader_getBootPackages () |
| #5 0x00000000400d41dc in java_lang_VMClassLoader__clinit_ () |
| #6 0x0000000000432848 in vm_class_init (vmc=0xbf1e58) at vm/class.c:717 |
| #7 0x000000000045c593 in vm_class_ensure_init (vmc=0xbf1e58) at include/vm/class.h:111 |
| #8 0x000000000045c857 in jit_magic_trampoline (cu=0xd5ac60) at jit/trampoline.c:104 |
| #9 0x00000000400d0a53 in java_lang_VMClassLoader_getSystemClassLoader_TP () |
| #10 0x00000000400d2b12 in java_lang_ClassLoader_StaticData__clinit_ () |
| #11 0x0000000000432848 in vm_class_init (vmc=0xbe7660) at vm/class.c:717 |
| #12 0x000000000044c051 in vm_class_ensure_init (vmc=0xbe7660) at include/vm/class.h:111 |
| #13 0x000000000044c544 in fixup_static_at (addr=1074593198) at arch/x86/fixup.c:137 |
| #14 0x0000000000441dc5 in static_field_signal_bh (ret=1074593198) at vm/static.c:66 |
| #15 0x000000000046b3b0 in signal_bh_trampoline () at arch/x86/signal-bh.S:124 |
| #16 0x00000000400cfdae in java_lang_ClassLoader_getSystemClassLoader () |
| #17 0x00000000004455d1 in native_call_gp (method=0xac01e0, target=0x400b6400, args=0x7fffffffd800) at arch/x86/call.c:206 |
| #18 0x0000000000445689 in native_call (method=0xac01e0, target=0x400b6400, args=0x7fffffffd800, result=0x7fffffffd890) |
| at arch/x86/call.c:257 |
| #19 0x000000000042fd52 in call_method_a (method=0xac01e0, target=0x400b6400, args=0x7fffffffd800, result=0x7fffffffd890) at vm/call.c:54 |
| #20 0x000000000042febc in vm_call_method_v (method=0xac01e0, args=0x7fffffffd8a0, result=0x7fffffffd890) at vm/call.c:71 |
| #21 0x00000000004345a5 in vm_call_method_object (method=0xac01e0) at include/vm/call.h:81 |
| #22 0x0000000000435f43 in get_system_class_loader () at vm/classloader.c:803 |
| #23 0x000000000041c649 in do_main_class () at vm/jato.c:941 |
| #24 0x000000000041cc77 in main (argc=7, argv=0x7fffffffdb48) at vm/jato.c:1144 |
| (gdb) |
| |
| As you can see, you must mangle method names before passing them to GDB: |
| |
| - Use the fully qualified name of the method. |
| |
| - Replace non-alphanumeric characters such as '/' and '.' with '_'. |
| |
| - This is true even for stuff like 'java/lang/VMClassLoader.<clinit>', which |
| becomes 'java_lang_VMClassLoader__clinit_'. |
| |
| - To refer to a method's trampoline, you append '_TP' at the end. |
| |
| There are a few caveats: |
| |
| - GDB issues those "not defined" warnings if the method hasn't been compiled |
| and registered yet. It's safe to ignore them. |
| |
| - If you set breakpoints they might stop working once you rerun Jato (by |
| using 'r' again). |
| |
| - Optimization CFLAGS may easily break GDB support. They are disabled by |
| DEBUG=1 itself, but adding your own CFLAGS might cause breakage. |
| |
| - Method arguments and line numbers aren't displayed. |
| |
| It may be useful to turn on "-Xtrace:jit" here as well, to get extra |
| information about what's going on while you're debugging. |
