Jmxtrans+influxdb+grafana monitoring zookeeper actual combat

  devops

Order

This article mainly studies how to monitor zookeeper using jmxtrans+influxdb+granfa.

Configure zookeeper jmx

Add zookeeper-env.sh to the conf directory and use chmod +x to grant execution permission, as follows

JMXLOCALONLY=false
JMXDISABLE=false
JMXPORT=8999
JMXAUTH=false
JMXSSL=false
  • Specify JMXPORT as 8999

Build influxdb and grafana

This is built using docker as follows

docker network create monitoring-network
docker run -d -p 8083:8083 -p 8086:8086 \
    --net monitoring-network  \
    --name influxdb  \
    influxdb:1.5.4
curl -POST http://192.168.99.100:8086/query --data-urlencode "q=CREATE DATABASE zookeeper"
docker run -d -p 3000:3000 --net monitoring-network --name grafana grafana/grafana:5.2.4
curl 'http://admin:admin@192.168.99.100:3000/api/datasources' -X POST \
-H 'Content-Type: application/json;charset=UTF-8' \
--data-binary '{"name":"influx","type":"influxdb","url":"http://192.168.99.100:8086",
"access":"proxy","isDefault":true,"database":"zookeeper","user":"admin","password":"admin"}'
  • Create a monitoring-network to connect the influxdb in the doc with grafana, and then use the rest api to create the zookeeper database.
  • Create grafana and specify net as monitoring-network and then configure the data source using rest api.

Jmxtrans configuration

  • Download jmxtrans
wget http://central.maven.org/maven2/org/jmxtrans/jmxtrans/270/jmxtrans-270-all.jar
  • Download the execution script
wget -q https://raw.githubusercontent.com/jmxtrans/jmxtrans/master/jmxtrans/jmxtrans.sh
chmod +x jmxtrans.sh
  • Json
{
  "servers": [
    {
      "port": 8999,
      "host": "localhost",
      "alias": "zk",
      "queries": [
        {
          "outputWriters" : [
            {
              "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
              "url" : "http://192.168.99.100:8086/",
              "database" : "zookeeper",
              "username" : "admin",
              "password" : "admin"
            } ],
          "obj": "org.apache.ZooKeeperService:name0=ReplicatedServer_id*",
          "attr": [
            "QuorumSize"
          ]
        },
        {
          "outputWriters" : [
            {
              "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
              "url" : "http://192.168.99.100:8086/",
              "database" : "zookeeper",
              "username" : "admin",
              "password" : "admin"
            } ],
          "obj": "org.apache.ZooKeeperService:name0=ReplicatedServer_id*,name1=replica*,name2=*",
          "attr": [
            "MaxClientCnxnsPerHost",
            "MaxRequestLatency",
            "AvgRequestLatency",
            "MinRequestLatency",
            "MaxSessionTimeout",
            "MinSessionTimeout",
            "OutstandingRequests",
            "PacketsReceived",
            "PacketsSent",
            "PendingRevalidationCount",
            "TickTime"
          ]
        },
        {
          "outputWriters" : [
            {
              "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
              "url" : "http://192.168.99.100:8086/",
              "database" : "zookeeper",
              "username" : "admin",
              "password" : "admin"
            } ],
          "obj": "org.apache.ZooKeeperService:name0=ReplicatedServer_id*,name1=replica*,name2=*,name3=InMemoryDataTree",
          "attr": [
            "NodeCount",
            "WatchCount"
          ]
        },
        {
          "outputWriters" : [
            {
              "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
              "url" : "http://192.168.99.100:8086/",
              "database" : "zookeeper",
              "username" : "admin",
              "password" : "admin"
            } ],
          "obj": "org.apache.ZooKeeperService:name0=ReplicatedServer_id*,name1=replica*,name2=*,name3=Connections,name4=*,name5=*",
          "resultAlias": "Clients",
          "attr": [
            "AvgLatency",
            "LastLatency",
            "MaxLatency",
            "MinLatency",
            "OutstandingRequests",
            "PacketsReceived",
            "PacketsSent"
          ]
        },
        {
          "outputWriters" : [
            {
              "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
              "url" : "http://192.168.99.100:8086/",
              "database" : "zookeeper",
              "username" : "admin",
              "password" : "admin"
            } ],
          "obj": "org.apache.ZooKeeperService:name0=StandaloneServer_port*",
          "resultAlias": "Clients",
          "attr": [
            "AvgLatency",
            "LastLatency",
            "MaxLatency",
            "MinLatency",
            "OutstandingRequests",
            "PacketsReceived",
            "PacketsSent",
            "NumAliveConnections"
          ]
        },
        {
          "outputWriters" : [
            {
              "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
              "url" : "http://192.168.99.100:8086/",
              "database" : "zookeeper",
              "username" : "admin",
              "password" : "admin"
            } ],
          "obj": "java.lang:type=Memory",
          "resultAlias": "Memory",
          "attr": [
            "HeapMemoryUsage",
            "NonHeapMemoryUsage"
          ]
        },
        {
          "outputWriters" : [
            {
              "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
              "url" : "http://192.168.99.100:8086/",
              "database" : "zookeeper",
              "username" : "admin",
              "password" : "admin"
            } ],
          "obj": "java.lang:type=OperatingSystem",
          "resultAlias": "Process",
          "attr": [
            "OpenFileDescriptorCount",
            "ProcessCpuLoad"
          ]
        },
        {
          "outputWriters" : [
            {
              "@class" : "com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory",
              "url" : "http://192.168.99.100:8086/",
              "database" : "zookeeper",
              "username" : "admin",
              "password" : "admin"
            } ],
          "obj": "java.lang:type=Threading",
          "resultAlias": "Threading",
          "attr": [
            "ThreadCount"
          ]
        }
      ],
      "numQueryThreads": 2
    }
  ]
}

Put it in the same directory as the jmxtrans.sh script and it can be loaded automatically.

  • Start up
JAR_FILE=jmxtrans-270-all.jar ./jmxtrans.sh start
  • Log output instance
2018-10-04 11:28:30 [main] level org.quartz.impl.StdSchedulerFactory [StdSchedulerFactory.java:1179] - Using default implementation for ThreadExecutor
2018-10-04 11:28:30 [main] level org.quartz.core.SchedulerSignalerImpl [SchedulerSignalerImpl.java:60] - Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl
2018-10-04 11:28:30 [main] level org.quartz.core.QuartzScheduler [QuartzScheduler.java:229] - Quartz Scheduler v.1.8.6 created.
2018-10-04 11:28:30 [main] level org.quartz.simpl.RAMJobStore [RAMJobStore.java:139] - RAMJobStore initialized.
2018-10-04 11:28:30 [main] level org.quartz.core.QuartzScheduler [QuartzScheduler.java:255] - Scheduler meta-data: Quartz Scheduler (v1.8.6) 'ServerScheduler' with instanceId 'Mars.local1538623710548'
  Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally.
  NOT STARTED.
  Currently in standby mode.
  Number of jobs executed: 0
  Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 10 threads.
  Using job-store 'org.quartz.simpl.RAMJobStore' - which does not support persistence. and is not clustered.

2018-10-04 11:28:30 [main] level org.quartz.impl.StdSchedulerFactory [StdSchedulerFactory.java:1324] - Quartz scheduler 'ServerScheduler' initialized from an externally opened InputStream.
2018-10-04 11:28:30 [main] level org.quartz.impl.StdSchedulerFactory [StdSchedulerFactory.java:1328] - Quartz scheduler version: 1.8.6
2018-10-04 11:28:30 [main] level org.quartz.core.QuartzScheduler [QuartzScheduler.java:2267] - JobFactory set to: com.googlecode.jmxtrans.guice.GuiceJobFactory@263f04ca
2018-10-04 11:28:30 [main] level com.googlecode.jmxtrans.JmxTransformer [JmxTransformer.java:177] - Starting Jmxtrans on : .
2018-10-04 11:28:30 [main] level org.quartz.core.QuartzScheduler [QuartzScheduler.java:519] - Scheduler ServerScheduler_$_Mars.local1538623710548 started.
2018-10-04 11:28:30 [main] level com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory [InfluxDbWriterFactory.java:121] - Result Tags to write set to: [ResultAttribute(name=typeName), ResultAttribute(name=objDomain), ResultAttribute(name=className), ResultAttribute(name=attributeName)]
2018-10-04 11:28:30 [main] level com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory [InfluxDbWriterFactory.java:102] - Connecting to url: http://192.168.99.100:8086/ as: username: admin
2018-10-04 11:28:31 [main] level com.googlecode.jmxtrans.model.output.InfluxDbWriterFactory [InfluxDbWriterFactory.java:121] - Result Tags to write set to: [ResultAttribute(name=typeName), ResultAttribute(name=objDomain), ResultAttribute(name=className), ResultAttribute(name=attributeName)]

Query influxdb data

docker exec -it influxdb influx
Connected to http://localhost:8086 version 1.5.4
InfluxDB shell version: 1.5.4
> use zookeeper
Using database zookeeper
> show MEASUREMENTS
name: measurements
name
----
Clients
Memory
Process
Threading
> show series from "Memory"
key
---
Memory,attributeName=HeapMemoryUsage,className=sun.management.MemoryImpl,hostname=zk,objDomain=java.lang,typeName=type\=Memory
Memory,attributeName=NonHeapMemoryUsage,className=sun.management.MemoryImpl,hostname=zk,objDomain=java.lang,typeName=type\=Memory
  • After that, specific index data can be viewed through select * from “Memory “
  • After the data is found, the configuration can be visualized on grafana and then monitored.

Summary

For applications such as zookeeper and kafka, jmx is built in, so its monitoring can be conducted through jmxtrans for index collection and conversion, and then output to influxdb or graphite or prometheus, etc. Finally, visualization and monitoring alarm can be conducted through grafana.

doc