Micrometer Prometheus

Prometheus is a dimensional time series database with a built-in UI, a custom query language, and math operations. Prometheus is designed to operate on a pull model, periodically scraping metrics from application instances, based on service discovery.

Micrometer uses the Prometheus Java Client under the hood; there are two versions of it and Micrometer supports both. If you want to use the "new" client (1.x), use micrometer-registry-prometheus but if you want to use the "legacy" client (0.x), use micrometer-registry-prometheus-simpleclient.

1. Installing micrometer-registry-prometheus

For Gradle, add the following implementation:

implementation 'io.micrometer:micrometer-registry-prometheus:latest.release'

For Maven, add the following dependency:

<dependency>
  <groupId>io.micrometer</groupId>
  <artifactId>micrometer-registry-prometheus</artifactId>
  <version>${micrometer.version}</version>
</dependency>

2. Installing micrometer-registry-prometheus-simpleclient

For Gradle, add the following implementation:

implementation 'io.micrometer:micrometer-registry-prometheus-simpleclient:latest.release'

For Maven, add the following dependency:

<dependency>
  <groupId>io.micrometer</groupId>
  <artifactId>micrometer-registry-prometheus-simpleclient</artifactId>
  <version>${micrometer.version}</version>
</dependency>

3. Configuring

Prometheus expects to scrape or poll individual application instances for metrics. In addition to creating a Prometheus registry, you also need to expose an HTTP endpoint to Prometheus’s scraper. In a Spring Boot application, a Prometheus actuator endpoint is auto-configured in the presence of Spring Boot Actuator. Otherwise, you can use any JVM-based HTTP server implementation to expose scrape data to Prometheus.

The following example uses the JDK’s com.sun.net.httpserver.HttpServer to expose a scrape endpoint:

PrometheusMeterRegistry prometheusRegistry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT);

try {
    HttpServer server = HttpServer.create(new InetSocketAddress(8080), 0);
    server.createContext("/prometheus", httpExchange -> {
        String response = prometheusRegistry.scrape(); (1)
        httpExchange.sendResponseHeaders(200, response.getBytes().length);
        try (OutputStream os = httpExchange.getResponseBody()) {
            os.write(response.getBytes());
        }
    });

    new Thread(server::start).start();
}
catch (IOException e) {
    throw new RuntimeException(e);
}
1 The PrometheusMeterRegistry has a scrape() function that knows how to supply the String data necessary for the scrape. All you have to do is wire it to an endpoint.

If you use the "new" client (micrometer-registry-prometheus), you can alternatively use io.prometheus.metrics.exporter.httpserver.HTTPServer, which you can find in io.prometheus:prometheus-metrics-exporter-httpserver (you need to add it as a dependency if you want to use it):

PrometheusMeterRegistry prometheusRegistry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT);
HTTPServer.builder()
    .port(8080)
    .registry(prometheusRegistry.getPrometheusRegistry())
    .buildAndStart();

If you use the "legacy" client (micrometer-registry-prometheus-simpleclient), you can alternatively use io.prometheus.client.exporter.HTTPServer, which you can find in io.prometheus:simpleclient_httpserver:

PrometheusMeterRegistry prometheusRegistry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT);
// you can set the daemon flag to false if you want the server to block
new HTTPServer(new InetSocketAddress(8080), prometheusRegistry.getPrometheusRegistry(), true);

If you use the "new" client (micrometer-registry-prometheus), another alternative can be io.prometheus.metrics.exporter.servlet.jakarta.PrometheusMetricsServlet, which you can find in io.prometheus:prometheus-metrics-exporter-servlet-jakarta in case your app is running in a servlet container (such as Tomcat):

PrometheusMeterRegistry prometheusRegistry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT);
HttpServlet servlet = new PrometheusMetricsServlet(prometheusRegistry.getPrometheusRegistry());

If you use the "legacy" client (micrometer-registry-prometheus-simpleclient), another alternative can be io.prometheus.client.exporter.MetricsServlet, which you can find in io.prometheus:simpleclient_servlet in case your app is running in a servlet container (such as Tomcat):

PrometheusMeterRegistry prometheusRegistry = new PrometheusMeterRegistry(PrometheusConfig.DEFAULT);
HttpServlet servlet = new MetricsServlet(prometheusRegistry.getPrometheusRegistry());

3.1. Scrape Format

By default, the PrometheusMeterRegistry scrape() method returns the Prometheus text format.

The OpenMetrics format can also be produced. To specify the format to be returned, you can pass a content type to the scrape method. For example, to get the OpenMetrics 1.0.0 format scrape, you could use the Prometheus Java client constant for it, as follows in case of the "new" client (micrometer-registry-prometheus):

String openMetricsScrape = registry.scrape("application/openmetrics-text");

if you use the "legacy" client (micrometer-registry-prometheus-simpleclient):

String openMetricsScrape = registry.scrape(TextFormat.CONTENT_TYPE_OPENMETRICS_100);

In Spring Boot applications, the Prometheus Actuator endpoint supports scraping in either format, defaulting to the Prometheus text format in the absence of a specific Accept header.

3.2. The Prometheus Rename Filter

In some cases, Micrometer provides instrumentation that overlaps with the commonly used Prometheus simple client modules but has chosen a different naming scheme for consistency and portability. If you wish to use the Prometheus "standard" names, add the following filter:

prometheusRegistry.config().meterFilter(new PrometheusRenameFilter());

3.3. Prometheus Client Properties

If you use the "new" client (micrometer-registry-prometheus), you can use some of the properties that the Prometheus Java Client supports, see the Prometheus Java Client Config docs. These properties can be loaded from any source that is supported by the Prometheus Java Client (Properties file, System properties, etc.) or they can be obtained through Micrometer using PrometheusConfig:

PrometheusConfig config = new PrometheusConfig() {
    @Override
    public String get(String key) {
        return null;
    }

    @Override
    public Properties prometheusProperties() {
        Properties properties = new Properties();
        properties.putAll(PrometheusConfig.super.prometheusProperties()); (1)
        properties.setProperty("io.prometheus.exemplars.sampleIntervalMilliseconds", "1"); (2)
        return properties;
    }
};

PrometheusMeterRegistry registry = new PrometheusMeterRegistry(config, new PrometheusRegistry(), Clock.SYSTEM);
1 You can reuse the "default" properties defined in PrometheusConfig.
2 You can set any property from any property source.

Micrometer passes these properties to the Exporters and the Examplar Sampler of the Prometheus client so you can use the exporter-, and the exemplar properties of the Prometheus Client.

4. Graphing

This section serves as a quick start to rendering useful representations in Prometheus for metrics originating in Micrometer. See the Prometheus docs for a far more complete reference of what is possible in Prometheus.

4.1. Grafana Dashboard

A publicly available Grafana dashboard for Micrometer-sourced JVM and Tomcat metrics is available here.

Grafana dashboard for JVM and Tomcat binders

The dashboard features:

  • JVM memory

  • Process memory (provided by micrometer-jvm-extras)

  • CPU-Usage, Load, Threads, File Descriptors, and Log Events

  • JVM Memory Pools (Heap, Non-Heap)

  • Garbage Collection

  • Classloading

  • Direct/Mapped buffer sizes

Instead of using the job tag to distinguish different applications, this dashboard makes use of a common tag called application, which is applied to every metric. You can apply the common tag, as follows:

registry.config().commonTags("application", "MYAPPNAME");

In Spring Boot applications, you can use the property support for common tags:

management.metrics.tags.application=MYAPPNAME

4.2. Counters

The query that generates a graph for the random-walk counter is rate(counter[10s]).

Grafana-rendered Prometheus counter
Figure 1. A Grafana rendered graph of the random walk counter.

Representing a counter without rate normalization over some time window is rarely useful, as the representation is a function of both the rapidity with which the counter is incremented and the longevity of the service. It is generally most useful to rate-normalize these time series to reason about them. Since Prometheus keeps track of discrete events across all time, it has the advantage of allowing for the selection of an arbitrary time window across which to normalize at query time (for example, rate(counter[10s]) provides a notion of requests per second over 10 second windows). The rate-normalized graph in the preceding image would return back to a value around 55 as soon as the new instance (say on a production deployment) was in service.

Grafana-rendered Prometheus counter (no rate)
Figure 2. Counter over the same random walk, no rate normalization.

In contrast, without rate normalization, the counter drops back to zero on service restart, and the count increases without bound for the duration of the service’s uptime.

4.3. Timers

The Prometheus Timer produces two counter time series with different names:

  • ${name}_count: Total number of all calls.

  • ${name}_sum: Total time of all calls.

Again, representing a counter without rate normalization over some time window is rarely useful, as the representation is a function of both the rapidity with which the counter is incremented and the longevity of the service.

Using the following Prometheus queries, we can graph the most commonly used statistics about timers:

  • Average latency: rate(timer_sum[10s])/rate(timer_count[10s])

  • Throughput (requests per second): rate(timer_count[10s])

Grafana-rendered Prometheus timer
Figure 3. Timer over a simulated service.

4.4. Long task timers

The following example shows a Prometheus query to plot the duration of a long task timer for a serial task is long_task_timer_sum. In Grafana, we can set an alert threshold at some fixed point.

Grafana-rendered Prometheus long task timer
Figure 4. Simulated back-to-back long tasks with a fixed alert threshold.

5. Limitation on same name with different set of tag keys

The PrometheusMeterRegistry doesn’t allow to create meters having the same name with a different set of tag keys, so you should guarantee that meters having the same name have the same set of tag keys. Otherwise, subsequent meters having the same name with a different set of tag keys will not be registered silently by default. You can change the default behavior by registering a meter registration failed listener. For example, you can register a meter registration failed listener that throws an exception as follows:

registry.config().onMeterRegistrationFailed((id, reason) -> {
    throw new IllegalArgumentException(reason);
});

Actually, the PrometheusMeterRegistry has a shortcut for this, so you can do the following to achieve the same:

registry.throwExceptionOnRegistrationFailure();

6. Exemplars

Exemplars are metadata that you can attach to the value of your time series. They can reference data outside of your metrics. A common use case is storing tracing information (traceId, spanId). Exemplars are not tags/dimensions (labels in Prometheus terminology), they will not increase cardinality since they belong to the values of the time series.

In order to setup Exemplars for PrometheusMeterRegistry, you will need a component that provides you the tracing information. If you use the "new" client (micrometer-registry-prometheus), this component is the io.prometheus.metrics.tracer.common.SpanContext while if you use the "legacy" client (micrometer-registry-prometheus-simpleclient), it is the SpanContextSupplier.

Setting them up are somewhat similar, if you use the "new" client (micrometer-registry-prometheus):

PrometheusMeterRegistry registry = new PrometheusMeterRegistry(
    PrometheusConfig.DEFAULT,
    new PrometheusRegistry(),
    Clock.SYSTEM,
    new MySpanContext() (1)
);
registry.counter("test").increment();
System.out.println(registry.scrape("application/openmetrics-text"));
1 You need to implement MySpanContext (class MySpanContext implements SpanContext { …​ }) or use an implementation that already exists.

But if you use the "legacy" client (micrometer-registry-prometheus-simpleclient):

PrometheusMeterRegistry registry = new PrometheusMeterRegistry(
    PrometheusConfig.DEFAULT,
    new CollectorRegistry(),
    Clock.SYSTEM,
    new DefaultExemplarSampler(new MySpanContextSupplier()) (1)
);
registry.counter("test").increment();
System.out.println(registry.scrape(TextFormat.CONTENT_TYPE_OPENMETRICS_100));
1 You need to implement MySpanContextSupplier (class MySpanContextSupplier implements SpanContextSupplier { …​ }) or use an implementation that already exists.

If your configuration is correct, you should get something like this, the # {span_id="321",trace_id="123"} …​ section is the Exempar right after the value:

# TYPE test counter
# HELP test
test_total 1.0 # {span_id="321",trace_id="123"} 1.0 1713310767.908
# EOF

Exemplars are only supported in the OpenMetrics format (they will not show up in the Prometheus text format). You might need to explicitly ask for the OpenMetrics format, for example:

curl --silent -H 'Accept: application/openmetrics-text; version=1.0.0' localhost:8080/prometheus