Jvm troubleshooting toolbox jvm-tools

  jvm

Order

This article mainly introduces a jvm troubleshooting toolbox: jvm-tools. In addition to encapsulating the basic jvm, jmx access and generation of flame maps are also provided.

download

Example

cpu and memory usage

java -jar sjk-0.9.jar ttop -o CPU -n 10 -p pid
java -jar sjk-0.9.jar ttop -o USER -n 10 -p pid
java -jar sjk-0.9.jar ttop -o SYS -n 10 -p pid
java -jar sjk-0.9.jar ttop -o ALLOC -n 10 -p pid

One command fixes the thread that caused the cpu to be high

root@d3daf1eca8ca:/# java -jar sjk-0.9.jar ttop -o CPU -n 10 -p 1
Monitoring threads ...

2018-01-01T07:18:26.773+0000 Process summary
  process cpu=6.88%
  application cpu=1.37% (user=1.30% sys=0.07%)
  other: cpu=5.51%
  thread count: 25
  GC time=0.00% (young=0.00%, old=0.00%)
  heap allocation rate 390kb/s
  safe point rate: 1.5 (events/s) avg. safe point pause: 0.16ms
  safe point sync time: 0.01% processing time: 0.02% (wallclock time)
[000068] user= 1.30% sys= 0.02% alloc=  386kb/s - RMI TCP Connection(1)-172.17.0.2
[000070] user= 0.00% sys= 0.03% alloc=  4268b/s - JMX server connection timeout 70
[000047] user= 0.00% sys= 0.01% alloc=     0b/s - NioBlockingSelector.BlockPoller-1
[000060] user= 0.00% sys= 0.01% alloc=    52b/s - http-nio-8080-AsyncTimeout
[000058] user= 0.00% sys= 0.00% alloc=    23b/s - http-nio-8080-ClientPoller-0
[000014] user= 0.00% sys= 0.00% alloc=     0b/s - container-0
[000002] user= 0.00% sys= 0.00% alloc=     0b/s - Reference Handler
[000003] user= 0.00% sys= 0.00% alloc=     0b/s - Finalizer
[000004] user= 0.00% sys= 0.00% alloc=     0b/s - Signal Dispatcher
[000013] user= 0.00% sys= 0.00% alloc=     0b/s - ContainerBackgroundProcessor[StandardEngine[Tomcat]]

histo

java -jar sjk-0.9.jar hh -n 10 --dead -p pid
java -jar sjk-0.9.jar hh -n 10 --dead-young -p pid
java -jar sjk-0.9.jar hh -n 10 --live -p pid

View object statistics

root@d3daf1eca8ca:/# java -jar sjk-0.9.jar hh -n 10 --live -p 1
   #      Instances          Bytes  Type
   1:         67025        9944432  [C
   2:          9382        2209656  [I
   3:         18413        1620344  java.lang.reflect.Method
   4:         65810        1579440  java.lang.String
   5:          4140        1283520  [B
   6:         35735        1143520  java.util.concurrent.ConcurrentHashMap$Node
   7:          8514         946936  java.lang.Class
   8:         15267         610680  java.util.LinkedHashMap$Entry
   9:          9027         505512  java.util.LinkedHashMap
  10:         10414         494656  [Ljava.lang.Object;
Total        475830       28175752

gc

java -jar sjk-0.9.jar gc -p pid

View gc log

root@d3daf1eca8ca:/# java -jar sjk-0.9.jar gc -p 1
MBean server connected
Collecting GC stats ...
[GC: Copy#112 time: 4ms mem: Survivor Space: 0k+135k->135k[max:17088k] Compressed Class Space: 5190k+0k->5190k[max:1048576k] Eden Space: 18496k-18496k->0k[max:137152k] Metaspace: 41247k+0k->41247k Tenured Gen: 27515k+0k->27515k[max:342720k]]
[GC: Copy#113 time: 2ms interval: 6649ms mem: Survivor Space: 135k+304k->439k[max:17088k,rate:45.83kb/s] Compressed Class Space: 5243k+0k->5243k[max:1048576k,rate:0.00kb/s] Eden Space: 18496k-18496k->0k[max:137152k,rate:-2781.77kb/s] Metaspace: 41696k+0k->41696k[rate:0.00kb/s] Tenured Gen: 27515k+0k->27515k[max:342720k,rate:0.00kb/s]]
[GC: Copy#114 time: 3ms interval: 675ms mem: Survivor Space: 439k+779k->1219k[max:17088k,rate:1154.48kb/s] Compressed Class Space: 5338k+0k->5338k[max:1048576k,rate:0.00kb/s] Eden Space: 18496k-18496k->0k[max:137152k,rate:-27401.48kb/s] Metaspace: 42470k+0k->42470k[rate:0.00kb/s] Tenured Gen: 27515k+0k->27515k[max:342720k,rate:0.00kb/s]]

stack

## collect 30 seconds
java -jar sjk-0.9.jar stcap -o dump.std -t 30000 -p pid
java -jar sjk-0.9.jar stcap -o dump.std -p pid

Dump thread stack

root@d3daf1eca8ca:/# java -jar sjk-0.9.jar stcap -o dump.std -p 1
Writing to /dump.std
Collected 506
Collected 1012
Collected 1518
Collected 2001
Collected 2507
//......
Trace dumped: 123211

View stacks or generate flame maps

java -jar sjk-0.9.jar ssa -f dump.std --print
java -jar sjk-0.9.jar ssa -f dump.std --histo
java -jar sjk-0.9.jar ssa -f dump.std --flame > flame.svg

图片描述

jmx

java -jar sjk-0.9.jar --help mx
java -jar sjk-0.9.jar mx -mi -b java.lang:type=Threading -p pid
java -jar sjk-0.9.jar mx -mg -b java.lang:type=Memory -f HeapMemoryUsage -p pid

View jmx

root@d3daf1eca8ca:/# java -jar sjk-0.9.jar mx -mg -b java.lang:type=Memory -f HeapMemoryUsage -p 1
java.lang:type=Memory
committed: 68194304
init:      33554432
max:       508887040
used:      49297880

Summary

It is very convenient to use ttop name to view threads with high cpu and to use stack stcap and ssa to generate flame maps, which is worth recommending.

doc