Talk about the jhsdb tool of openjdk.

  jdk, jvm

Order

This article mainly studies the jhsdb tool of openjdk.

sa-jdi.jar

export JAVA_HOME="/Library/Java/JavaVirtualMachines/jdk1.8.0_151.jdk/Contents/Home"
chmod +x $JAVA_HOME/lib/sa-jdi.jar
java -cp $JAVA_HOME/lib/sa-jdi.jar sun.jvm.hotspot.HSDB
java -cp $JAVA_HOME/lib/sa-jdi.jar sun.jvm.hotspot.CLHSDB
  • Before java9, there was an sa-jdi.jar in the JAVA_HOME/lib directory, which could be started by the above command (Graphical interface) and CLHSDB (command line)
  • Sa in sa-jdi.jar is fully called Service Abilities Agent. It was previously a component provided by sun Company to assist in debugging HotSpot, and HSDB is implemented using Service Abilities Agent.
  • HSDB is short for HotSpot Debugger, because the Serviceability Agent will attach the process first, then suspend the process for snapshot, and finally deattach the process (The process resumed operation.), so pay attention to when using HSDB

jhsdb

/ # jhsdb
    clhsdb           command line debugger
    debugd           debug server
    hsdb             ui debugger
    jstack --help    to get more information
    jmap   --help    to get more information
    jinfo  --help    to get more information
    jsnap  --help    to get more information
  • Jhsdb was introduced by java9 and can be found in JAVA_HOME/bin directory. It replaced JAVA_HOME/lib/sa-jdi.jar before jdk9.
  • Jhsdb has clhsdb, debugd, hsdb, jstack, jmap, jinfo, jsnap mode that can be used.
  • Where hsdb is ui debugger, which is sun.jvm.hotspot.HSDB; before jdk9; And clhsdb is sun.jvm.hotspot.CLHSDB before jdk9.

jhsdb jstack

/ # jhsdb jstack --help
    --locks    to print java.util.concurrent locks
    --mixed    to print both java and native frames (mixed mode)
    --exe    executable image name
    --core    path to coredump
    --pid    pid of process to attach

–pid is used to specify the process ID of JVM; –exe is used to specify an executable file; -corespecifies the core dump file.

abnormal

jhsdb jstack --mixed --pid 1
//......
Caused by: sun.jvm.hotspot.debugger.DebuggerException: get_thread_regs failed for a lwp
    at jdk.hotspot.agent/sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.getThreadIntegerRegisterSet0(Native Method)
    at jdk.hotspot.agent/sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1GetThreadIntegerRegisterSetTask.doit(LinuxDebuggerLocal.java:534)
    at jdk.hotspot.agent/sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:151)

If this exception indicates a problem with the jdk version, you can try other jdk compiled versions.

debugger

/ # jhsdb jstack --locks --pid 1
Attaching to process ID 1, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 12+33
Deadlock Detection:

No deadlocks found.

"DestroyJavaVM" #32 prio=5 tid=0x000055c3b5be0800 nid=0x6 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE
   JavaThread state: _thread_blocked

Locked ownable synchronizers:
    - None

"http-nio-8080-Acceptor-0" #30 daemon prio=5 tid=0x000055c3b5d71800 nid=0x2f runnable [0x00007fa0d13de000]
   java.lang.Thread.State: RUNNABLE
   JavaThread state: _thread_in_native
 - sun.nio.ch.ServerSocketChannelImpl.accept0(java.io.FileDescriptor, java.io.FileDescriptor, java.net.InetSocketAddress[]) @bci=0 (Interpreted frame)
 - sun.nio.ch.ServerSocketChannelImpl.accept(java.io.FileDescriptor, java.io.FileDescriptor, java.net.InetSocketAddress[]) @bci=4, line=525 (Interpreted frame)
 - sun.nio.ch.ServerSocketChannelImpl.accept() @bci=41, line=277 (Interpreted frame)
 - org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept() @bci=4, line=448 (Interpreted frame)
 - org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept() @bci=1, line=70 (Interpreted frame)
 - org.apache.tomcat.util.net.Acceptor.run() @bci=98, line=95 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=835 (Interpreted frame)

Locked ownable synchronizers:
    - <0x00000000e3aab6e0>, (a java/util/concurrent/locks/ReentrantLock$NonfairSync)

"http-nio-8080-ClientPoller-0" #29 daemon prio=5 tid=0x000055c3b5c20000 nid=0x2e runnable [0x00007fa0d14df000]
   java.lang.Thread.State: RUNNABLE
   JavaThread state: _thread_in_native
 - sun.nio.ch.EPoll.wait(int, long, int, int) @bci=0 (Interpreted frame)
 - sun.nio.ch.EPollSelectorImpl.doSelect(java.util.function.Consumer, long) @bci=96, line=120 (Interpreted frame)
 - sun.nio.ch.SelectorImpl.lockAndDoSelect(java.util.function.Consumer, long) @bci=42, line=124 (Interpreted frame)
    - locked <0x00000000e392ece8> (a sun.nio.ch.EPollSelectorImpl)
    - locked <0x00000000e392ee38> (a sun.nio.ch.Util$2)
 - sun.nio.ch.SelectorImpl.select(long) @bci=31, line=136 (Interpreted frame)
 - org.apache.tomcat.util.net.NioEndpoint$Poller.run() @bci=55, line=743 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=835 (Interpreted frame)

Locked ownable synchronizers:
    - None

"http-nio-8080-exec-10" #28 daemon prio=5 tid=0x000055c3b48d6000 nid=0x2d waiting on condition [0x00007fa0d15e0000]
   java.lang.Thread.State: WAITING (parking)
   JavaThread state: _thread_blocked
 - jdk.internal.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
    - parking to wait for <0x00000000e3901670> (a java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject)
 - java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=14, line=194 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() @bci=42, line=2081 (Interpreted frame)
 - java.util.concurrent.LinkedBlockingQueue.take() @bci=27, line=433 (Interpreted frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=36, line=107 (Interpreted frame)
 - org.apache.tomcat.util.threads.TaskQueue.take() @bci=1, line=33 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=147, line=1054 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1114 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=628 (Interpreted frame)
 - org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run() @bci=4, line=61 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=835 (Interpreted frame)
 //......

/ # jhsdb jstack --mixed --pid 1
Attaching to process ID 1, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 12+33
Deadlock Detection:

No deadlocks found.

----------------- 47 -----------------
"http-nio-8080-Acceptor-0" #30 daemon prio=5 tid=0x000055c3b5d71800 nid=0x2f runnable [0x00007fa0d13de000]
   java.lang.Thread.State: RUNNABLE
   JavaThread state: _thread_in_native
0x00007fa0ee0923ad        ????????
----------------- 46 -----------------
"http-nio-8080-ClientPoller-0" #29 daemon prio=5 tid=0x000055c3b5c20000 nid=0x2e runnable [0x00007fa0d14df000]
   java.lang.Thread.State: RUNNABLE
   JavaThread state: _thread_in_native
0x00007fa0ee05f3d0    epoll_pwait + 0x1d
0x00007fa0daa97810    * sun.nio.ch.EPoll.wait(int, long, int, int) bci:0 (Interpreted frame)
0x00007fa0daa91680    * sun.nio.ch.EPollSelectorImpl.doSelect(java.util.function.Consumer, long) bci:96 line:120 (Interpreted frame)
0x00007fa0db85f57c    * sun.nio.ch.SelectorImpl.lockAndDoSelect(java.util.function.Consumer, long) bci:42 line:124 (Compiled frame)
* sun.nio.ch.SelectorImpl.select(long) bci:31 line:136 (Compiled frame)
* org.apache.tomcat.util.net.NioEndpoint$Poller.run() bci:55 line:743 (Interpreted frame)
0x00007fa0daa91c88    * java.lang.Thread.run() bci:11 line:835 (Interpreted frame)
0x00007fa0daa88849    <StubRoutines>
0x00007fa0ed122952    _ZN9JavaCalls11call_helperEP9JavaValueRK12methodHandleP17JavaCallArgumentsP6Thread + 0x3c2
0x00007fa0ed1208d0    _ZN9JavaCalls12call_virtualEP9JavaValue6HandleP5KlassP6SymbolS6_P6Thread + 0x200
0x00007fa0ed1ccfc5    _ZL12thread_entryP10JavaThreadP6Thread + 0x75
0x00007fa0ed74f3a3    _ZN10JavaThread17thread_main_innerEv + 0x103
0x00007fa0ed74c3f5    _ZN6Thread8call_runEv + 0x75
0x00007fa0ed4a477e    _ZL19thread_native_entryP6Thread + 0xee
//......

–locks or –mixed may take longer (A few minutes, maybe nearly 6 minutes), so the process may be suspended for a long time, so pay attention to when using these two options

jhsdb jmap

jmap -heap pid

/ # jmap -heap 1
Error: -heap option used
Cannot connect to core dump or remote debug server. Use jhsdb jmap instead

When jdk9 and above used jmap -heap pid command to check the current heap usage, they found an error and indicated that jhsdb jmap was needed to replace it.

jhsdb jmap pid

/ # jhsdb jmap 1
sh: jhsdb: not found

When jlink was found, jdk.hotspot.agent was not added as the module. after this module was added, jhsdb was found in JAVA_HOME/bin directory.

PTRACE_ATTACH failed

/ # jhsdb jmap 1
You have to set --pid or --exe.
    <no option>    to print same info as Solaris pmap
    --heap    to print java heap summary
    --binaryheap    to dump java heap in hprof binary format
    --dumpfile    name of the dump file
    --histo    to print histogram of java object heap
    --clstats    to print class loader statistics
    --finalizerinfo    to print information on objects awaiting finalization
    --exe    executable image name
    --core    path to coredump
    --pid    pid of process to attach
/ # jhsdb jmap --heap --pid 1
Attaching to process ID 1, please wait...
ERROR: ptrace(PTRACE_ATTACH, ..) failed for 1: Operation not permitted
Error attaching to process: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process: ptrace(PTRACE_ATTACH, ..) failed for 1: Operation not permitted
sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process: ptrace(PTRACE_ATTACH, ..) failed for 1: Operation not permitted
    at jdk.hotspot.agent/sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.execute(LinuxDebuggerLocal.java:176)
    at jdk.hotspot.agent/sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach(LinuxDebuggerLocal.java:336)
    at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.attachDebugger(HotSpotAgent.java:672)
    at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.setupDebuggerLinux(HotSpotAgent.java:612)
    at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.setupDebugger(HotSpotAgent.java:338)
    at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:305)
    at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.attach(HotSpotAgent.java:141)
    at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.start(Tool.java:185)
    at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
    at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.main(JMap.java:176)
    at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.runJMAP(SALauncher.java:326)
    at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.main(SALauncher.java:455)
Caused by: sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the process: ptrace(PTRACE_ATTACH, ..) failed for 1: Operation not permitted
    at jdk.hotspot.agent/sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach0(Native Method)
    at jdk.hotspot.agent/sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$1AttachTask.doit(LinuxDebuggerLocal.java:326)
    at jdk.hotspot.agent/sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal$LinuxDebuggerLocalWorkerThread.run(LinuxDebuggerLocal.java:151)

It was found that PTRACE_ATTACH is disabled by docker and PTRACE_ATTACH needs to be enabled when the container is running.

Docker enables SYS_PTRACE

docker run --cap-add=SYS_PTRACE

After that, jhsdb can be used normally as follows:

/ # jhsdb jmap --heap --pid 1
Attaching to process ID 1, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 12+33

using thread-local object allocation.
Shenandoah GC with 4 thread(s)

Heap Configuration:
   MinHeapFreeRatio         = 40
   MaxHeapFreeRatio         = 70
   MaxHeapSize              = 523763712 (499.5MB)
   NewSize                  = 1363144 (1.2999954223632812MB)
   MaxNewSize               = 17592186044415 MB
   OldSize                  = 5452592 (5.1999969482421875MB)
   NewRatio                 = 2
   SurvivorRatio            = 8
   MetaspaceSize            = 21807104 (20.796875MB)
   CompressedClassSpaceSize = 1073741824 (1024.0MB)
   MaxMetaspaceSize         = 17592186044415 MB
   ShenandoahRegionSize     = 262144 (0.25MB)

Heap Usage:
Shenandoah Heap:
   regions   = 1997
   capacity  = 523501568 (499.25MB)
   used      = 70470552 (67.2059555053711MB)
   committed = 144441344 (137.75MB)

jhsdb jinfo

/ # jhsdb jinfo --help
    --flags    to print VM flags
    --sysprops    to print Java System properties
    <no option>    to print both of the above
    --exe    executable image name
    --core    path to coredump
    --pid    pid of process to attach

Sysprops that display jinfo using jhsdb are as follows:

/ # jhsdb jinfo --sysprops --pid 1
Attaching to process ID 1, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 12+33
awt.toolkit = sun.awt.X11.XToolkit
java.specification.version = 12
sun.jnu.encoding = UTF-8
//......

This command is actually equivalent to jinfo -sysprops 1.

jhsdb jsnap

/ # jhsdb jsnap --pid 1
Attaching to process ID 1, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 12+33
java.threads.started=27 event(s)
java.threads.live=24
java.threads.livePeak=24
java.threads.daemon=20
java.cls.loadedClasses=8250 event(s)
java.cls.unloadedClasses=1 event(s)
java.cls.sharedLoadedClasses=0 event(s)
java.cls.sharedUnloadedClasses=0 event(s)
java.ci.totalTime=18236958158 tick(s)
java.property.java.vm.specification.version=12
java.property.java.vm.specification.name=Java Virtual Machine Specification
java.property.java.vm.specification.vendor=Oracle Corporation
java.property.java.vm.version=12+33
java.property.java.vm.name=OpenJDK 64-Bit Server VM
java.property.java.vm.vendor=Azul Systems, Inc.
java.property.java.vm.info=mixed mode
java.property.jdk.debug=release
//......

Jhsdb jsnap’s function is mainly provided by sun.jvm.hotspot.tools.JSnap.java in jdk.hotspot.agent module. It can be used to view threads and class loading/unloading related event, JVM attribute parameters, etc., among which –all can display more JVM attribute parameters

Jhsdb and jcmd

jhsdb: A New Tool for JDK 9This article lists the equivalent commands of jhsdb and jcmd, as shown in the following figure:
图片描述

Summary

  • Before java9, there was an sa-jdi.jar in the JAVA_HOME/lib directory, which could be started by the above command (Graphical interface) and CLHSDB (command line); Sa in sa-jdi.jar is fully called Service Abilities Agent. It was previously a component provided by sun Company to assist in debugging HotSpot, and HSDB is implemented by using Service Abilities Agent. HSDB is short for HotSpot Debugger, because the Serviceability Agent will attach the process first, then suspend the process for snapshot, and finally deattach the process (The process resumed operation.), so pay attention to when using HSDB
  • Jhsdb was introduced by java9 and can be found in JAVA_HOME/bin directory. It replaced java _ home/lib/sa-jdi.jar before jdk9; Jhsdb has clhsdb, debugd, hsdb, jstack, jmap, jinfo, jsnap mode that can be used; Where hsdb is ui debugger, which is sun.jvm.hotspot.HSDB; before jdk9; And clhsdb is sun.jvm.hotspot.CLHSDB before jdk9.
  • Jhsdb is in jdk.hotspot.agent module; The –locks or –mixed command for jhsdb jstack may take a long time (A few minutes, maybe nearly 6 minutes), thus the process may be suspended for a long time, so pay attention to when using these two options; Jhdbjmap -heap-PID is required to replace jdk9 and later versions which no longer use jmap-heap command to query heap memory. Using jhsdb jmap requires PTRACE_ATTACH to be enabled when the container is running.

doc