Micrometer OTLP

OpenTelemetry is a CNCF incubating project for providing standards for telemetry data. OpenTelemetry protocol (OTLP) is a vendor-neutral protocol that you can use to send data to various backends that support it. You can read the corresponding docs on how the metrics are ingested and can be visualized in the respective vendor docs.

1. Installing micrometer-registry-otlp

For Gradle, add the following implementation:

implementation 'io.micrometer:micrometer-registry-otlp:latest.release'

For Maven, add the following dependency:

<dependency>
  <groupId>io.micrometer</groupId>
  <artifactId>micrometer-registry-otlp</artifactId>
  <version>${micrometer.version}</version>
</dependency>

2. Configuring

The following example configures an OTLP registry:

OtlpConfig otlpConfig = new OtlpConfig() {
            @Override
            public String get(final String key) {
                return null;
            }
};

MeterRegistry registry = new OtlpMeterRegistry(otlpConfig, Clock.SYSTEM);

OtlpConfig is an interface with a set of default methods. If, in the implementation of get(String k), rather than returning null, you instead bind it to a property source (for example, a simple Map can work), you can override the default configuration through properties. For example, Micrometer’s Spring Boot support binds properties prefixed with management.otlp.metrics.export directly to the OtlpConfig:

management:
  otlp:
    metrics:
      export:
        # Supported configs
        url: "https://otlp.example.com:4318/v1/metrics"
        batchSize: 15000
        aggregationTemporality: "cumulative"
        headers:
          header1: value1
        step: 30s
        resourceAttributes:
          key1: value1

  1. url - The URL to which data is reported. Environment variables OTEL_EXPORTER_OTLP_METRICS_ENDPOINT and OTEL_EXPORTER_OTLP_ENDPOINT are also supported in the default implementation. If a value is not provided, it defaults to localhost:4318/v1/metrics

  2. batchSize - number of Meter to include in a single payload sent to the backend. The default is 10,000.

  3. aggregationTemporality - Aggregation temporality determines how the additive quantities are expressed, in relation to time. The environment variable OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE is supported by the default implementation. The supported values are cumulative or delta. Defaults to cumulative.

  4. headers - Additional headers to send with exported metrics. This can be used for authorization headers. By default, headers are loaded from the config. If that is not set, they can be taken from the environment variables OTEL_EXPORTER_OTLP_HEADERS and OTEL_EXPORTER_OTLP_METRICS_HEADERS. If a header is set in both the environmental variables, the header in the latter overrides the former.

  5. step - the interval at which metrics will be published. The environment variable OTEL_METRIC_EXPORT_INTERVAL is supported by the default implementation. If a value is not provided, defaults to 1 minute.

  6. resourceAttributes - Resource attributes are used for all metrics published. By default, Micrometer adds the following resource attributes:

Key Default value

telemetry.sdk.name

io.micrometer

telemetry.sdk.language

java

telemetry.sdk.version

<micrometer-core-version> (e.g.: 1.11.0)

service.name

unknown_service

If this config is empty, resource attributes are loaded from the OTEL_RESOURCE_ATTRIBUTES environmental variable. You can override service.name by setting the OTEL_SERVICE_NAME environmental variable, and this takes precedence over other configs.

3. Supported metrics

Metric points define the different data points that are supported in OTLP. Micrometer supports exporting the below data points in OTLP format,

  1. Sums

  2. Gauge

  3. Histogram

  4. Exponential Histogram

  5. Summary

The following table maps OTLP data points and the Micrometer meters:

OTLP data point Micrometer meter type

Sums

Counter, FunctionCounter

Gauge

Gauge, TimeGauge, MultiGauge

Histogram

Timer, DistributionSummary, LongTaskTimer, FunctionTimer (only sum and count are set)

Exponential Histogram*

Timer, DistributionSummary

Summary

Timer, DistributionSummary, LongTaskTimer

* - Histogram or Exponential Histogram is determined based on the configuration of the OtlpMeterRegistry. See below section for more info.

Note:

  1. max on the Histogram data point is supported only in delta aggregation temporality. This is because the values represented by cumulative min and max stabilize as more events are recorded and are less useful when recorded over application’s lifecycle.

  2. Currently, Micrometer only exports metadata for type Meter to OTLP.

4. Histograms and Percentiles

Micrometer Timer and DistributionSummary support configuring client-side percentiles and percentile histograms. The OTLP specification terms the Summary data point (client-side percentiles) as legacy and not recommended for new applications. The Summary data point also cannot have min/max associated with it. Due to these reasons, Micrometer prefers exporting Timers and DistributionSummary as a Histogram data point. By default, a Timer/DistributionSummary without any additional percentile/histogram config is exported as a Histogram data point. However, by configuring the timer to generate only client-side percentiles, by using publishPercentiles, you can change this to a Summary data point that exports pre-calculated percentiles. When both publishPercentiles and (publishPercentileHistogram or serviceLevelObjectives) are configured, the Histogram data point is preferred and pre-calculated percentiles are not generated. The following table describes which data point is used with different configurations:

Configuration OTLP data point

publishPercentiles

Summary

publishPercentileHistogram

Histogram/Exponential Histogram *

publishPercentiles and publishPercentileHistogram

Histogram/Exponential Histogram *

serviceLevelObjectives

Histogram

publishPercentiles and serviceLevelObjectives

Histogram

* The configuration histogramFlavor determines whether the OTLP DataPoint is a Histogram/Exponential Histogram.

OtlpMeterRegistry supports 2 types of Histogram implementations (1.Explicit Bucket Histogram (or simply called Histogram), 2. Exponential Histogram) when publishPercentileHistogram is configured. The choice is chosen by setting histogramFlavor in OtlpConfig used by registry. When the implementation is exponential histogram, it also supports 2 additional properties

  1. maxScale used to cap the maximum scale used by the Exponential Histogram (defaults to 20).

  2. maxBuckets determines the maximum number of buckets to be used for exponential histograms (defaults to 160).

Since Exponential Histogram cannot have custom SLO’s specified, explicit bucket histogram is used whenever serviceLevelObjectives are added.

4.1. Configuration with Spring Boot

If you use Spring Boot, you can use the per-meter properties to configure this behavior.

If you want to generate a Histogram data point for a Timer with name test.timer and default buckets generated by Micrometer, use:

management.metrics.distribution.percentiles-histogram.test.timer=true

For buckets with customized SLO, use:

management.metrics.distribution.slo.test.timer=10.0,100.0,500.0,1000.0

Alternatively, if you want to generate Summary data point for a timer with name test.timer and 90th and 99th percentiles, you can use:

management.metrics.distribution.percentiles.test.timer=0.9,0.99