Talk about micrometer’s HistogramGauges

  springboot

Order

This article mainly studies the HistogramGauges of micrometer.

AutoConfiguration

For springboot applications, it is equipped with various export AutoConfiguration packages, as shown in the org.Spring Framework.boot.action.autoconfiguration.metrics.export package. The following types of export are currently supported in version 2.0.1:

atlas、datadog、ganglia、graphite、influx、jmx、newrelic、prometheus、properties、signalfx、simple、statsd、wavefront

Let’s look at the AutoConfiguration of statsd and prometheus.

StatsdMetricsExportAutoConfiguration

spring-boot-actuator-autoconfigure-2.0.1.RELEASE-sources.jar! /org/springframework/boot/actuate/autoconfigure/metrics/export/statsd/StatsdMetricsExportAutoConfiguration.java

@Configuration
@AutoConfigureBefore({ CompositeMeterRegistryAutoConfiguration.class,
        SimpleMetricsExportAutoConfiguration.class })
@AutoConfigureAfter(MetricsAutoConfiguration.class)
@ConditionalOnBean(Clock.class)
@ConditionalOnClass(StatsdMeterRegistry.class)
@ConditionalOnProperty(prefix = "management.metrics.export.statsd", name = "enabled", havingValue = "true", matchIfMissing = true)
@EnableConfigurationProperties(StatsdProperties.class)
public class StatsdMetricsExportAutoConfiguration {

    @Bean
    @ConditionalOnMissingBean
    public StatsdConfig statsdConfig(StatsdProperties statsdProperties) {
        return new StatsdPropertiesConfigAdapter(statsdProperties);
    }

    @Bean
    @ConditionalOnMissingBean
    public StatsdMeterRegistry statsdMeterRegistry(StatsdConfig statsdConfig,
            Clock clock) {
        return new StatsdMeterRegistry(statsdConfig, clock);
    }

    @Bean
    public StatsdMetrics statsdMetrics() {
        return new StatsdMetrics();
    }

}

As you can see, StatsdMeterRegistry was created

PrometheusMetricsExportAutoConfiguration

spring-boot-actuator-autoconfigure-2.0.1.RELEASE-sources.jar! /org/springframework/boot/actuate/autoconfigure/metrics/export/prometheus/PrometheusMetricsExportAutoConfiguration.java

@Configuration
@AutoConfigureBefore({ CompositeMeterRegistryAutoConfiguration.class,
        SimpleMetricsExportAutoConfiguration.class })
@AutoConfigureAfter(MetricsAutoConfiguration.class)
@ConditionalOnBean(Clock.class)
@ConditionalOnClass(PrometheusMeterRegistry.class)
@ConditionalOnProperty(prefix = "management.metrics.export.prometheus", name = "enabled", havingValue = "true", matchIfMissing = true)
@EnableConfigurationProperties(PrometheusProperties.class)
public class PrometheusMetricsExportAutoConfiguration {

    @Bean
    @ConditionalOnMissingBean
    public PrometheusConfig prometheusConfig(PrometheusProperties prometheusProperties) {
        return new PrometheusPropertiesConfigAdapter(prometheusProperties);
    }

    @Bean
    @ConditionalOnMissingBean
    public PrometheusMeterRegistry prometheusMeterRegistry(
            PrometheusConfig prometheusConfig, CollectorRegistry collectorRegistry,
            Clock clock) {
        return new PrometheusMeterRegistry(prometheusConfig, collectorRegistry, clock);
    }

    @Bean
    @ConditionalOnMissingBean
    public CollectorRegistry collectorRegistry() {
        return new CollectorRegistry(true);
    }

    @ManagementContextConfiguration
    public static class PrometheusScrapeEndpointConfiguration {

        @Bean
        @ConditionalOnEnabledEndpoint
        @ConditionalOnMissingBean
        public PrometheusScrapeEndpoint prometheusEndpoint(
                CollectorRegistry collectorRegistry) {
            return new PrometheusScrapeEndpoint(collectorRegistry);
        }

    }

}

You can see that PrometheusMeterRegistry was created

Timer.register

micrometer-core-1.0.3-sources.jar! /io/micrometer/core/instrument/Timer.java

        /**
         * Add the timer to a single registry, or return an existing timer in that registry. The returned
         * timer will be unique for each registry, but each registry is guaranteed to only create one timer
         * for the same combination of name and tags.
         *
         * @param registry A registry to add the timer to, if it doesn't already exist.
         * @return A new or existing timer.
         */
        public Timer register(MeterRegistry registry) {
            // the base unit for a timer will be determined by the monitoring system implementation
            return registry.timer(new Meter.Id(name, tags, null, description, Type.TIMER), distributionConfigBuilder.build(),
                    pauseDetector == null ? registry.config().pauseDetector() : pauseDetector);
        }

You can see that the register is delegated to the registry.timer method

MeterRegistry

micrometer-core-1.0.3-sources.jar! /io/micrometer/core/instrument/MeterRegistry.java

    /**
     * Only used by {@link Timer#builder(String)}.
     *
     * @param id                          The identifier for this timer.
     * @param distributionStatisticConfig Configuration that governs how distribution statistics are computed.
     * @return A new or existing timer.
     */
    Timer timer(Meter.Id id, DistributionStatisticConfig distributionStatisticConfig, PauseDetector pauseDetectorOverride) {
        return registerMeterIfNecessary(Timer.class, id, distributionStatisticConfig, (id2, filteredConfig) -> {
            Meter.Id withUnit = id2.withBaseUnit(getBaseTimeUnitStr());
            return newTimer(withUnit, filteredConfig.merge(defaultHistogramConfig()), pauseDetectorOverride);
        }, NoopTimer::new);
    }

    /**
     * Build a new timer to be added to the registry. This is guaranteed to only be called if the timer doesn't already exist.
     *
     * @param id                          The id that uniquely identifies the timer.
     * @param distributionStatisticConfig Configuration for published distribution statistics.
     * @param pauseDetector               The pause detector to use for coordinated omission compensation.
     * @return A new timer.
     */
    protected abstract Timer newTimer(Meter.Id id, DistributionStatisticConfig distributionStatisticConfig, PauseDetector pauseDetector);

The newTimer abstract method is called here

StatsdMeterRegistry.newTimer

micrometer-registry-statsd-1.0.3-sources.jar! /io/micrometer/statsd/StatsdMeterRegistry.java

    @SuppressWarnings("ConstantConditions")
    @Override
    protected Timer newTimer(Meter.Id id, DistributionStatisticConfig distributionStatisticConfig, PauseDetector
            pauseDetector) {
        Timer timer = new StatsdTimer(id, lineBuilder(id), publisher, clock, distributionStatisticConfig, pauseDetector, getBaseTimeUnit(),
                statsdConfig.step().toMillis());
        HistogramGauges.registerWithCommonFormat(timer, this);
        return timer;
    }

It can be seen that histogram gauges. registerwithcommonformat (timer, this) was called in the newTimer operation.

HistogramGauges.registerWithCommonFormat

micrometer-core-1.0.3-sources.jar! /io/micrometer/core/instrument/distribution/HistogramGauges.java

    /**
     * Register a set of gauges for percentiles and histogram buckets that follow a common format when
     * the monitoring system doesn't have an opinion about the structure of this data.
     */
    public static HistogramGauges registerWithCommonFormat(Timer timer, MeterRegistry registry) {
        Meter.Id id = timer.getId();
        return HistogramGauges.register(timer, registry,
                percentile -> id.getName() + ".percentile",
                percentile -> Tags.concat(id.getTags(), "phi", DoubleFormat.decimalOrNan(percentile.percentile())),
                percentile -> percentile.value(timer.baseTimeUnit()),
                bucket -> id.getName() + ".histogram",
                bucket -> Tags.concat(id.getTags(), "le", DoubleFormat.decimalOrWhole(bucket.bucket(timer.baseTimeUnit()))));
    }

It can be seen that the registration is carried out here using HistogramGauges, the name of percentileName is id.getName()+”.percentile “,and the name of bucketName is id.getName()+”.histogram”

HistogramGauges

micrometer-core-1.0.3-sources.jar! /io/micrometer/core/instrument/distribution/HistogramGauges.java

    private HistogramGauges(HistogramSupport meter, MeterRegistry registry,
                            Function<ValueAtPercentile, String> percentileName,
                            Function<ValueAtPercentile, Iterable<Tag>> percentileTags,
                            Function<ValueAtPercentile, Double> percentileValue,
                            Function<CountAtBucket, String> bucketName,
                            Function<CountAtBucket, Iterable<Tag>> bucketTags) {
        this.meter = meter;

        HistogramSnapshot initialSnapshot = meter.takeSnapshot();
        this.snapshot = initialSnapshot;

        ValueAtPercentile[] valueAtPercentiles = initialSnapshot.percentileValues();
        CountAtBucket[] countAtBuckets = initialSnapshot.histogramCounts();

        this.totalGauges = valueAtPercentiles.length + countAtBuckets.length;

        // set to zero initially, so the first polling of one of the gauges on each publish cycle results in a
        // new snapshot
        this.polledGaugesLatch = new CountDownLatch(0);

        for (int i = 0; i < valueAtPercentiles.length; i++) {
            final int index = i;

            ToDoubleFunction<HistogramSupport> percentileValueFunction = m -> {
                snapshotIfNecessary();
                polledGaugesLatch.countDown();
                return percentileValue.apply(snapshot.percentileValues()[index]);
            };

            Gauge.builder(percentileName.apply(valueAtPercentiles[i]), meter, percentileValueFunction)
                    .tags(percentileTags.apply(valueAtPercentiles[i]))
                    .register(registry);
        }

        for (int i = 0; i < countAtBuckets.length; i++) {
            final int index = i;

            ToDoubleFunction<HistogramSupport> bucketCountFunction = m -> {
                snapshotIfNecessary();
                polledGaugesLatch.countDown();
                return snapshot.histogramCounts()[index].count();
            };

            Gauge.builder(bucketName.apply(countAtBuckets[i]), meter, bucketCountFunction)
                    .tags(bucketTags.apply(countAtBuckets[i]))
                    .register(registry);
        }
    }

It can be seen that here a percentileValues is taken for the HistogramSnapshot and a Gauge is registered, and then a corresponding Gauge is registered for the CountAtBucket[] of the HistogramSnapshot.

Example

SimpleMeterRegistry simpleMeterRegistry = new SimpleMeterRegistry();
    @Test
    public void testHistogramGauges() throws InterruptedException {
        Timer timer = Timer.builder("api-cost")
                .publishPercentileHistogram()
                .publishPercentiles(0.95,0.99)
                .register(simpleMeterRegistry);

        IntStream.rangeClosed(1,1000)
                .forEach(i -> {
                    timer.record(Duration.ofMillis(ThreadLocalRandom.current().nextInt(200)));
                    simpleMeterRegistry.getMeters()
                            .stream()
                            .forEach(m -> {
                                System.out.println(m.getId() + "-->" + m.measure());
                            });
                });
        TimeUnit.MINUTES.sleep(5);
    }

Output instance

MeterId{name='api-cost.percentile', tags=[ImmutableTag{key='phi', value='0.95'}]}-->[Measurement{statistic='VALUE', value=0.192905216}]
MeterId{name='api-cost.percentile', tags=[ImmutableTag{key='phi', value='0.99'}]}-->[Measurement{statistic='VALUE', value=0.201293824}]
MeterId{name='api-cost', tags=[]}-->[Measurement{statistic='COUNT', value=999.0}, Measurement{statistic='TOTAL_TIME', value=97.158}, Measurement{statistic='MAX', value=0.199}]
MeterId{name='api-cost.percentile', tags=[ImmutableTag{key='phi', value='0.95'}]}-->[Measurement{statistic='VALUE', value=0.192905216}]
MeterId{name='api-cost.percentile', tags=[ImmutableTag{key='phi', value='0.99'}]}-->[Measurement{statistic='VALUE', value=0.201293824}]
MeterId{name='api-cost', tags=[]}-->[Measurement{statistic='COUNT', value=1000.0}, Measurement{statistic='TOTAL_TIME', value=97.348}, Measurement{statistic='MAX', value=0.199}]

Summary

Currently, only Prometheus and Atlas support percentile histograms. However, micrometer simply supports Lower Percentile on the client side, but it is not as flexible as server side support and cannot aggregate across tags. Currently, tag is reported together as part of MetID. For the calculation of qps, it can be measured by Timer type and then counted by percentile index and group according to time interval.

doc