Secrets of building a debuggable runtime: Learn how language implementors solve your runtime issues

J9, OMR and beyond
Serviceable Runtimes

• Bjørn Vårdal
• IBM Runtimes Developer
• J9 JVM
• @bvaardal
• MethodHandles
• Lambdas
• VarHandles
• Project Valhalla
Bjørn Vårdal IBM Runtime Technologies 2
Introduction

Most die young_______runtimes
unless you service them.

The information contained in this presentation is provided for informational purposes only.
Whilst efforts were made to verify the completeness and accuracy of the information contained in this
presentation, it is provided “as is”, without warranty of any kind, express or implied.
All performance data included in this presentation have been gathered in a controlled environment. Your own test
results may vary based on hardware, software or infrastructure differences.
All data included in this presentation are meant to be used only as a guide.
In addition, the information contained in this presentation is based on IBM’s current product plans and strategy,
which are subject to change by IBM, without notice.
IBM and its affiliated companies shall not be responsible for any damages arising out of the use of, or otherwise
related to, this presentation or any other documentation.
Nothing contained in this presentation is intended to, or shall have the effect of:
- creating any warrant or representation from IBM, its affiliated companies or its or their suppliers and/or licensors
Disclaimer

• Advanced core dump analysis for J9
• Used by the J9 service team
• Used by J9 developers
• Front-end: DDR Interactive
Direct Dump Reader

$ java -cp j9ddr.jar com.ibm.j9ddr.tools.ddrinteractive.DDRInteractive core.12345
DDR Interactive
Ships with J9

DDR Interactive

*0 : PID: 2824; !j9javavm 0x7f27480eef0
Run !j9help to see a list of commands
>
DDR Interactive

>
DDR Interactive

> !j9help
DDR Interactive

> !j9help
threads Lists VM threads
stack Walks the Java stack for <thread>
stackslots Walks the Java stack (including objects) for <thread>
.
.
.
classforname Find the class corresponding to name (with wildcards)
dumpromclass Dump the specified J9ROMClass. Wild cards are allowed in class name.
findpattern Search memory for a specific pattern
whatis Determine the type of structure at a given address
DDR - !j9help

$ java -cp j9ddr.jar com.ibm.j9ddr.tools.ddrinteractive.DDRInteractive core.20170315.010552.2824.01.dmp
> !threads
DDR - !threads

> !threads
!stack 0x0228d70 !j9vmthread 0x0228d70 !j9thread 0x07f274807d70 tid 0x4b (75) // (main)
!stack 0x02294f0 !j9vmthread 0x02294f0 !j9thread 0x7f27480b0bf8 tid 0x4e (78) // (JIT Compilation Thread-1)
!stack 0x022af20 !j9vmthread 0x022af20 !j9thread 0x7f27480bfcb0 tid 0x55 (85) // (JIT-SamplerThread)
!stack 0x022b2e0 !j9vmthread 0x022b2e0 !j9thread 0x7f27480c0228 tid 0x56 (86) // (IProfiler)
!stack 0x0238620 !j9vmthread 0x0238620 !j9thread 0x7f2748238170 tid 0x57 (87) // (Common-Cleaner)
!stack 0x023f480 !j9vmthread 0x023f480 !j9thread 0x7f27482386e8 tid 0x58 (88) // (Signal Dispatcher)
!stack 0x023f840 !j9vmthread 0x023f840 !j9thread 0x7f27483d01f8 tid 0x5a (90) // (Concurrent Mark Helper)
!stack 0x0023fc0 !j9vmthread 0x0023fc0 !j9thread 0x7f27483d0c90 tid 0x5b (91) // (GC Slave)
!stack 0x0002420 !j9vmthread 0x0002420 !j9thread 0x007f26e084d0 tid 0x5f (95) // (Attach API wait loop)
!stack 0x0240730 !j9vmthread 0x0240730 !j9thread 0x7f27483d2218 tid 0x60 (96) // (Finalizer thread)
>
DDR - !threads

> !threads
>
DDR - !threads

> !stack 0x0228d70
<1a2970> Generic special frame
<1a2970> !j9method 0x01B9A718 X.print(Ljava/lang/String;)V
<1a2970> !j9method 0x01B9A6F8 X.m(I)V
<1a2970> !j9method 0x01B9A6D8 X.main([Ljava/lang/String;)V
<1a2970> JNI call-in frame
<1a2970> Native method frame
>
DDR - !stack

> !stackslots 0x0228d70
<1a2970> *** BEGIN STACK WALK, flags = 0401 walkThread = 0x01A2970 ***
<1a2970> ITERATE_O_SLOTS
<1a2970> RECORD_BYTECODE_PC_OFFSET
<1a2970> Initial values: walkSP = 0x01AE6D60, PC = 0x01, literals = 0x0, A0 = 0x01AE6D78, j2iFrame = 0x0, ...
<1a2970> Generic special frame: bp = 0x01AE6D78, sp = 0x01AE6D60, pc = 0x01, cp = 0x0, arg0EA = 0x01AE ...
<1a2970> Bytecode frame: bp = 0x01AE6D90, sp = 0x01AE6D80, pc = 0x07FDCB43A6344, cp = 0x01B9A580, arg0EA = 0x01AE6D98, ...
<1a2970> Method: X.print(Ljava/lang/String;)V !j9method 0x01B9A718
<1a2970> Bytecode index = 16
<1a2970> Using local mapper
<1a2970> Locals starting at 0x01AE6D98 for 0x01 slots
<1a2970> I-Slot: a0[0x01AE6D98] = 0x0FFEF46D0
<1a2970> Bytecode frame: bp = 0x01AE6DB0, sp = 0x01AE6DA0, pc = 0x07FDCB43A630A, cp = 0x01B9A580, arg0EA = 0x01AE6DB8, ...
<1a2970> Method: X.m(I)V !j9method 0x01B9A6F8
<1a2970> Locals starting at 0x01AE6DB8 for 0x01 slots
<1a2970> I-Slot: a0[0x01AE6DB8] = 0x01234
<1a2970> Bytecode frame: bp = 0x01AE6DD0, sp = 0x01AE6DC0, pc = 0x07FDCB43A62DB, cp = 0x1B9A580, arg0EA = 0x1AE6DD8, ...
<1a2970> Method: X.main([Ljava/lang/String;)V !j9method 0x01B9A6D8
<1a2970> Locals starting at 0x01AE6DD8 for 0x01 slots
<1a2970> I-Slot: a0[0x01AE6DD8] = 0x0FFEE5668
<1a2970> JNI call-in frame: bp = 0x01AE6E0, sp = 0x01AE6DE0, pc = 0x07FDCB96DE314, cp = 0x0, arg0EA = 0x01AE6E0, ...
<1a2970> New ELS = 0x0
<1a2970> JNI native method frame: bp = 0x01AE6E88, sp = 0x01AE6E08, pc = 0x07, cp = 0x0, arg0EA = 0x01AE6E88, ...
<1a2970> Object pushes starting at 0x01AE6E08 for 12 slots
DDR - !stackslots

<1a2970> I-Slot: a0[0x01AE6DB8] = 0x01234
<1a2970> New ELS = 0x0
DDR - !stackslots

> !j9object 0x0FFEF46D0
J9VMJavaLangString at 0x00000000FFEF46D0 {
struct J9Class* clazz = !j9class 0x1A5B500 // java/lang/String
Object flags = 0x00000000;
[B value = !fj9object 0x86f36828 (offset=0) (java/lang/String)
I count = 0x00000007 (offset=4) (java/lang/String)
I hashCode = 0x00000000 (offset=8) (java/lang/String)
"foo4660"
}
DDR - !j9object

GDB
• Requires intimate knowledge
about the JVM internal structures
• Requires debug symbols
• Source must match
• Platform specific
DDR
• Has intimate knowledge about
the JVM internal structures
• Doesn’t require debug symbols
• Cross JVM version
• Cross platform
GDB vs DDR

<1a2970> Initial values: walkSP = 0x01AE6D60, A0 = 0x01AE6D78,
<1a2970> Generic special frame: bp = 0x01AE6D78, sp = 0x01AE6D60,
<1a2970> Bytecode frame: bp = 0x01AE6D90, sp = 0x01AE6D80, arg0EA =
<1a2970> Bytecode frame: bp = 0x01AE6DB0, sp = 0x01AE6DA0, pc =
<1a2970> I-Slot: a0[0x01AE6DB8] = 0x01234
<1a2970> Bytecode frame: bp = 0x01AE6DD0, arg0EA = 0x1AE6DD8
<1a2970> JNI call-in frame: sp = 0x01AE6DE0, pc = 0x07FDCB96DE314
<1a2970> JNI native method frame: cp = 0x0, arg0EA = 0x01AE6E88, ...
(gdb) x/200 0x0000000001AE6D60
0x1ae6d60: 0x00000000 0x00000000 0x01b9a718 0x00000000
0x1ae6d70: 0xb43a6344 0x00007fdc 0x01ae6d9a 0x00000000
0x1ae6d80: 0x01b9a6f8 0x00000000 0xb43a630a 0x00007fdc
0x1ae6d90: 0x01ae6db8 0x00000000 0xffef46d0 0x00000000
0x1ae6da0: 0x01b9a6d8 0x00000000 0xb43a62db 0x00007fdc
0x1ae6db0: 0x01ae6dd8 0x00000000 0x00001234 0x00000000
0x1ae6dc0: 0x00000000 0x00000000 0xb96de314 0x00007fdc
0x1ae6dd0: 0x01ae6e00 0x00000000 0xffee5668 0x00000000
0x1ae6de0: 0x00000000 0x00000000 0x00020000 0x00000000
0x1ae6df0: 0x00000060 0x00000000 0x00000007 0x00000000
0x1ae6e00: 0x01ae6e8a 0x00000000 0xffee5668 0x00000000
0x1ae6e10: 0x86ef0828 0x00000000 0x86f317d0 0x00000000
0x1ae6e20: 0x86f317d0 0x00000000 0x00000000 0x00000000
0x1ae6e30: 0x00000000 0x00000000 0x86f2ed50 0x00000000
0x1ae6e40: 0x86ef3548 0x00000000 0x86ef34b8 0x00000000
0x1ae6e50: 0x86ef0240 0x00000000 0x86ef0fb0 0x00000000
DDR
GDB vs DDR
GDB

DDR Interactive
”The tool that I use the most and that saves me the most time.”
J9 Service Team Lead
“We can implement commands to look for known symptoms,
which drastically reduces our triage work.”
J9 Service Team Member

• We are open-sourcing the J9 JVM
• DDR coming along for the ride 
• http://openj9.org
OpenJ9

Reliable Shared Components for High Performance Language Runtimes
http://www.eclipse.org/omr
https://github.com/eclipse/omr
https://developer.ibm.com/open/omr
Eclipse OMR

Agenda
• DDR in J9
• Capabilities
• How DDR is implemented
• OMR ddrgen
• Generating the DDR resources
• DDR in a new runtime
• Adding DDR capabilities to Base9
• Loading a Base9 core into DDR Interactive

• Print structure at address
!j9object 0x12345678
!j9method 0xABCDEF00
• Other printing commands
!stackslots 0xABCDEF0
• Utility functions
!classforname
!findpattern
!whatis
• Data analysis
gccheck
deadlock
What can DDR do?
>

• Print structure at address
!j9object 0x12345678
!j9method 0xABCDEF00
• Other printing commands
!stackslots 0xABCDEF0
• Utility functions
!classforname
!findpattern
!whatis
• Data analysis
gccheck
deadlock
What can DDR do?
> gccheck
Starting GC Check
Checking HEAP...done (919 ms).
Checking CLASS HEAP...done (296 ms).
Checking REMEMBERED SET...done (54 ms).
Checking UNFINALIZED...done (3 ms).
Checking FINALIZABLE...done (3 ms).
Checking OWNABLE_SYNCHRONIZER...done (1 ms).
Checking STRING TABLE...done (516 ms).
Checking CLASS LOADERS...done (4 ms).
Checking JNI GLOBAL REFS...done (1 ms).
Checking JNI WEAK GLOBAL REFS...done (1 ms).
Checking JVMTI OBJECT TAG TABLES...done (3 ms).
Checking VM CLASS SLOTS...done (1 ms).
Checking MONITOR TABLE...done (7 ms).
Checking VM THREAD SLOTS...done (208 ms).
Checking THREAD STACKS...done (25 ms).
Done (2058ms)

• Structure layout information in
the process memory
• DDR data generator generates
blob and code
• Detectable RAS structure in
memory at runtime
How does DDR work?
Runtime
DDR InteractiveDDR data
generator
DDR
blob
Core dump
DDR
blob
Generated
code
RAS RAS

DDR Components
OpenJ9Eclipse OMR
J9 JVM
DDR data
generator
DDR Interactive
RAS

RAS Structure
Runtime memory
VM structure
Threads
Port library
etc.

RAS Structure
Runtime memory
RAS Structure
Eye catcher: “J9VMRAS”
Bit pattern: 0xAA55AA55
...
DDR Structure blob
…
VM
…
VM structure
Threads
Port library
etc.

RAS Structure
Runtime memory
DDR blob
Structure layouts
Field offsets
Build flags
RAS Structure
...
DDR Structure blob
…
VM
…
VM structure
Threads
Port library
etc.

DDR Structure Blob
Runtime memory
DDR blob
Structure layouts
Field offsets
Build flags
RAS Structure
...
DDR Structure blob
…
VM
…
VM structure
Threads
Port library
etc.

OMR ddrgen
Compile
with debug
symbols
ddrgen
blob.dat
Load into
memory at
runtime
Scan source
files for
build flags
./getMacros.sh .
libdwarf

DDR Class Generation
superset.out
DDR Interactive
(j9ddr.jar)
Generate Java
structure classes
superset.out
Generate Java
pointer classes

DDR Interactive startup
Generated
Java classes
DDR Interactive
DDR blob
from core file
DDR
ClassLoader
Template Field offsets
Load structure
classes

• DDR capabilities – Analyzing the JVM in a core file
• RAS structure – A detectable starting point
• OMR ddrgen – Creating the DDR blob and superset
• DDR Interactive – Using the blob and superset
Summary

DDR in a new Runtime

• Small tutorial runtime with ~10 opcodes
• For illustrating basic OMR JITBuilder functionality
• For demonstrating basic DDR functionality?
• My fork: https://github.com/bjornvar/Base9/tree/ddr
Base9

• Quick prototype
• ~1 hour of hacking
• Mostly copied from J9 JVM
DDR in Base9

RAS Structure in J9
JVM runtime memory
DDR blob
Structure layouts
Build flags
RAS Structure
...
DDR Structure blob
…
JavaVM
…
JavaVM structure
Threads
Port library
etc.

RAS Structure in Base9
Base9 runtime memory
DDR blob
Structure layouts
Build flags
Generated by running
ddrgen on Base9
RAS Structure
...
DDR Structure blob
…
ExecutionContext
Copied the structure
from the J9 source
ExecutionContext
Stack
String table
etc.

Base9 Code Changes

RAS Structure in Base9
Base9 runtime memory
DDR blob
Structure layouts
Build flags
RAS Structure
...
DDR Structure blob
…
ExecutionContext
ExecutionContext
Stack
String table
etc.

DDR Interactive
Some changes were required:
• Disabled all J9 specific functionality (!stackslots, etc.)
• Told it to look for B9RAS instead of J9RAS
• Added build flags constants

$
Bjørn Vårdal IBM Runtimes 52
DDR Interactive

What’s next?
• Make DDR Interactive plug-and-play
• Always support structure commands on DDR enabled core files
• Easy DDR generation process
• Release DDR Interactive and DDR generators
• OMR
• OpenJ9

Conclusion
• DDR in J9
• Capabilities
• How DDR is implemented
• OMR ddrgen
• Turning debug symbols into blob and superset
• DDR in Base9
• Adding DDR capabilities to a new runtime
• Loading a Base9 core into DDR Interactive

Q & A

Secrets of building a debuggable runtime: Learn how language implementors solve your runtime issues

Recommended

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Secrets of building a debuggable runtime: Learn how language implementors solve your runtime issues (20)

More from Dev_Events (20)

Recently uploaded (20)

Secrets of building a debuggable runtime: Learn how language implementors solve your runtime issues