GDB is great for stepping through machine code to figure out what is going on. It uses debug information under the hood to present you with a tidy backtrace and also determine how much machine code to print when you type disassemble.

This debug information comes from your compiler. Clang, GCC, rustc, etc all produce debug data in a format called DWARF and then embed that debug information inside the binary (ELF, Mach-O, …) when you do -ggdb or equivalent.

Unfortunately, this means that by default, GDB has no idea what is going on if you break in a JIT-compiled function. You can step instruction-by-instruction and whatnot, but that’s about it. This is because the current instruction pointer is nowhere to be found in any of the existing debug info tables from the host runtime code, so your terminal is filled with ???. See this example from the V8 docs:

#8  0x08281674 in v8::internal::Runtime_SetProperty (args=...) at src/runtime.cc:3758
#9  0xf5cae28e in ?? ()
#10 0xf5cc3a0a in ?? ()
#11 0xf5cc38f4 in ?? ()
#12 0xf5cbef19 in ?? ()
#13 0xf5cb09a2 in ?? ()
#14 0x0809e0a5 in v8::internal::Invoke (...) at src/execution.cc:97

Fortunately, there is a JIT interface to GDB. If you implement a couple of functions in your JIT and run them every time you finish compiling a function, you can get the debugging niceties for your JIT code too. See again a V8 example:

#6  0x082857fc in v8::internal::Runtime_SetProperty (args=...) at src/runtime.cc:3758
#7  0xf5cae28e in ?? ()
#8  0xf5cc3a0a in loop () at test.js:6
#9  0xf5cc38f4 in test.js () at test.js:13
#10 0xf5cbef19 in ?? ()
#11 0xf5cb09a2 in ?? ()
#12 0x0809e1f9 in v8::internal::Invoke (...) at src/execution.cc:97

Unfortunately, the GDB docs are somewhat sparse. So I went spelunking through a bunch of different projects to try and understand what is going on.

The big picture (and the old interface)

GDB expects your runtime to expose a function called __jit_debug_register_code and a global variable called __jit_debug_descriptor. GDB automatically adds its own internal breakpoints at this function, if it exists. Then, when you compile code, you call this function from your runtime.

In slightly more detail:

1. Compile a function in your JIT compiler. This gives you a function name, maybe other metadata, an executable code address, and a code size
2. Generate an entire ELF/Mach-O/… object in-memory (!) for that one function, describing its name, code region, maybe other DWARF metadata such as line number maps
3. Write a jit_code_entry linked list node that points at your object (“symfile”)
4. Link it into the __jit_debug_descriptor linked list
5. Call __jit_debug_register_code, which gives GDB control of the process so it can pick up the new function’s metadata
6. Optionally, break into (or crash inside) one of your JITed functions
7. At some point, later, when your function gets GCed, unregister your code by editing the linked list and calling __jit_debug_register_code again

This is why you see compiler projects such as V8 including large swaths of code just to make object files:

• V8

• Cinder

• Zend PHP

• CoreCLR/.NET

• QEMU

• JavaScriptCore

• LuaJIT

• ART

• which looks like it does something smart about grouping the JIT code entries together (RepackEntries), but I’m not sure exactly what it does

• HHVM

• TomatoDotNet

• Jato JVM

• a minimal example

• monoruby

• Mono

• It looks like Dart used to have support for this but has since removed it

Because this is a huge hassle, GDB also has a newer interface that does not require making an ELF/Mach-O/…+DWARF object.

Custom debug info (the new interface)

This new interface requires writing a binary format of your choice. You make the writer and you make the reader. Then, when you are in GDB, you load your reader as a shared object.

Here are some details from Sanjoy Das.

Only a few runtimes implement this interface:

• yk write yk read

• asmjit-utilities write asmjit-utilities read

• Erlang/OTP write

• It looks like they do write their own custom debug format but I can’t find the reader plugin

• FEX write FEX read

I think it also requires at least the reader to proclaim it is GPL via the macro GDB_DECLARE_GPL_COMPATIBLE_READER.

Since I wrote about the perf map interface recently, I have it on my mind. Why can’t we use it in GDB?

Adapting to the Linux perf interface

I suppose it would be possible to try and upstream a patch to GDB to support the Linux perf map interface for JITs. After all, why shouldn’t it be able to automatically pick up symbols from /tmp/perf-...? That would be great baseline debug info for “free”.

In the meantime, maybe it is reasonable to create a re-usable custom debug reader:

• When registering code, write the address and name to /tmp/perf-... as you normally would
• Write the filename as the symfile (does this make /tmp the magic number?)
• Have the debug info reader just parse the perf map file

It would be less flexible than both the DWARF and custom readers support: it would only be able to handle filename and code region. No embedding source code for GDB to display in your debugger. But maybe that is okay for a partial solution?

The n-squared problem

V8 notes in their GDB JIT docs that because the JIT interface is a linked list and we only keep a pointer to the head, we get O(n2) behavior. Bummer. This becomes especially noticeable since they register additional code objects not just for functions, but also trampolines, cache stubs, etc.

Garbage collection

Since GDB expects the code pointer in your symbol object file not to move, you have to make sure to have a stable symbol file pointer and stable executable code pointer. To make this happen, V8 disables its moving GC.

Additionally, if your compiled function gets collected, you have to make sure to unregister the function. Instead of doing this eagerly, ART treats the GDB JIT linked list as a weakref and periodically removes dead code entries from it.