Observation Components
In this section we will describe main components related to Micrometer Observation.
Micrometer Observation basic flow
Observation through ObservationRegistry gets created with a mutable Observation.Context. On each Micrometer Observation lifecycle action (e.g. start()) a corresponding ObservationHandler method is called (e.g. onStart) with the mutable Observation.Context as argument.
┌───────────────────┐┌───────┐
│ObservationRegistry││Context│
└┬──────────────────┘└┬──────┘
┌▽────────────────────▽┐
│Observation │
└┬─────────────────────┘
┌▽──────┐
│Handler│
└───────┘Micrometer Observation detailed flow
┌───────────────────┐┌───────┐┌─────────────────────┐┌────────────────────┐
│ObservationRegistry││Context││ObservationConvention││ObservationPredicate│
└┬──────────────────┘└┬──────┘└┬────────────────────┘└┬───────────────────┘
┌▽────────────────────▽────────▽──────────────────────▽┐
│Observation │
└┬─────────────────────────────────────────────────────┘
┌▽──────┐
│Handler│
└┬──────┘
┌▽────────────────┐
│ObservationFilter│
└─────────────────┘Observation through ObservationRegistry gets created with a mutable Observation.Context. To allow name and key-value customization, an ObservationConvention can be used instead of direct name setting. List of ObservationPredicate is run to verify if an Observation should be created instead of a no-op version. On each Micrometer Observation lifecycle action (e.g. start()) a corresponding ObservationHandler method is called (e.g. onStart) with the mutable Observation.Context as argument. On Observation stop, before calling the ObservationHandler onStop methods, list of ObservationFilter is called to optionally further modify the Observation.Context.
Observation.Context
To pass information between the instrumented code and the handler (or between handler methods, such as onStart and onStop), you can use an Observation.Context. An Observation.Context is a Map-like container that can store values for you while your handler can access the data inside the context.
Observation Handler
Observation Handler allows adding capabilities to existing instrumentations (i.e. you instrument code once and depending on the Observation Handler setup, different actions, such as create spans, metrics, logs will happen). In other words, if you have instrumented code and want to add metrics around it, it’s enough for you to register an Observation Handler in the Observation Registry to add that behaviour.
Let’s look at the following example of adding a timer behaviour to an existing instrumentation.
A popular way to record Observations is storing the start state in a Timer.Sample instance and stopping it when the event has ended.
Recording such measurements could look like this:
MeterRegistry registry = new SimpleMeterRegistry();
Timer.Sample sample = Timer.start(registry);
try {
// do some work here
}
finally {
sample.stop(Timer.builder("my.timer").register(registry));
}If you want to have more observation options (such as metrics and tracing — already included in Micrometer — plus anything else you will plug in), you need to rewrite that code to use the Observation API.
ObservationRegistry registry = ObservationRegistry.create();
Observation.createNotStarted("my.operation", registry).observe(this::doSomeWorkHere);Starting with Micrometer 1.10, you can register "handlers" (ObservationHandler instances) that are notified about the lifecycle event of an observation (for example, you can run custom code when an observation is started or stopped).
Using this feature lets you add tracing capabilities to your existing metrics instrumentation (see: DefaultTracingObservationHandler). The implementation of these handlers does not need to be tracing related. It is completely up to you how you are going to implement them (for example, you can add logging capabilities).
ObservationHandler Example
Based on this, we can implement a simple handler that lets the users know about its invocations by printing them out to stdout:
static class SimpleHandler implements ObservationHandler<Observation.Context> {
@Override
public void onStart(Observation.Context context) {
System.out.println("START " + "data: " + context.get(String.class));
}
@Override
public void onError(Observation.Context context) {
System.out.println("ERROR " + "data: " + context.get(String.class) + ", error: " + context.getError());
}
@Override
public void onEvent(Observation.Event event, Observation.Context context) {
System.out.println("EVENT " + "event: " + event + " data: " + context.get(String.class));
}
@Override
public void onStop(Observation.Context context) {
System.out.println("STOP " + "data: " + context.get(String.class));
}
@Override
public boolean supportsContext(Observation.Context handlerContext) {
// you can decide if your handler should be invoked for this context object or
// not
return true;
}
}You need to register the handler to the ObservationRegistry:
ObservationRegistry registry = ObservationRegistry.create();
registry.observationConfig().observationHandler(new SimpleHandler());You can use the observe method to instrument your codebase:
ObservationRegistry registry = ObservationRegistry.create();
Observation.Context context = new Observation.Context().put(String.class, "test");
// using a context is optional, so you can call createNotStarted without it:
// Observation.createNotStarted(name, registry)
Observation.createNotStarted("my.operation", () -> context, registry).observe(this::doSomeWorkHere);You can also take full control of the scoping mechanism:
ObservationRegistry registry = ObservationRegistry.create();
Observation.Context context = new Observation.Context().put(String.class, "test");
// using a context is optional, so you can call start without it:
// Observation.start(name, registry)
Observation observation = Observation.start("my.operation", () -> context, registry);
try (Observation.Scope scope = observation.openScope()) {
doSomeWorkHere();
}
catch (Exception ex) {
observation.error(ex); // and don't forget to handle exceptions
throw ex;
}
finally {
observation.stop();
}Signaling Errors and Arbitrary Events
When instrumenting code, we might want to signal that an error happened or signal that an arbitrary event happened. The observation API lets us do so through its error and event methods.
One use-case for signaling an arbitrary event can be attaching annotations to Span for Distributed Tracing, but you can also process them however you want in your own handler, such as emitting log events based on them:
ObservationRegistry registry = ObservationRegistry.create();
Observation observation = Observation.start("my.operation", registry);
try (Observation.Scope scope = observation.openScope()) {
observation.event(Observation.Event.of("my.event", "look what happened"));
doSomeWorkHere();
}
catch (Exception exception) {
observation.error(exception);
throw exception;
}
finally {
observation.stop();
}Observation.ObservationConvention Example
When instrumenting code, we want to provide sensible defaults for tags, but also we want to let users easily change those defaults. An ObservationConvention interface is a description of what tags and name we should create for an Observation.Context:
/**
* A dedicated {@link Observation.Context} used for taxing.
*/
class TaxContext extends Observation.Context {
private final String taxType;
private final String userId;
TaxContext(String taxType, String userId) {
this.taxType = taxType;
this.userId = userId;
}
String getTaxType() {
return taxType;
}
String getUserId() {
return userId;
}
}
/**
* An example of an {@link ObservationFilter} that will add the key-values to all
* observations.
*/
class CloudObservationFilter implements ObservationFilter {
@Override
public Observation.Context map(Observation.Context context) {
return context.addLowCardinalityKeyValue(KeyValue.of("cloud.zone", CloudUtils.getZone()))
.addHighCardinalityKeyValue(KeyValue.of("cloud.instance.id", CloudUtils.getCloudInstanceId()));
}
}
/**
* An example of an {@link ObservationConvention} that renames the tax related
* observations and adds cloud related tags to all contexts. When registered via the
* `ObservationRegistry#observationConfig#observationConvention` will override the
* default {@link TaxObservationConvention}. If the user provides a custom
* implementation of the {@link TaxObservationConvention} and passes it to the
* instrumentation, the custom implementation wins.
*
* In other words
*
* 1) Custom {@link ObservationConvention} has precedence 2) If no custom convention
* was passed and there's a matching {@link GlobalObservationConvention} it will be
* picked 3) If there's no custom, nor matching global convention, the default
* {@link ObservationConvention} will be used
*
* If you need to add some key-values regardless of the used
* {@link ObservationConvention} you should use an {@link ObservationFilter}.
*/
class GlobalTaxObservationConvention implements GlobalObservationConvention<TaxContext> {
// this will be applicable for all tax contexts - it will rename all the tax
// contexts
@Override
public boolean supportsContext(Observation.Context context) {
return context instanceof TaxContext;
}
@Override
public String getName() {
return "global.tax.calculate";
}
}
// Interface for an ObservationConvention related to calculating Tax
interface TaxObservationConvention extends ObservationConvention<TaxContext> {
@Override
default boolean supportsContext(Observation.Context context) {
return context instanceof TaxContext;
}
}
/**
* Default convention of tags related to calculating tax. If no user one or global
* convention will be provided then this one will be picked.
*/
class DefaultTaxObservationConvention implements TaxObservationConvention {
@Override
public KeyValues getLowCardinalityKeyValues(TaxContext context) {
return KeyValues.of(TAX_TYPE.withValue(context.getTaxType()));
}
@Override
public KeyValues getHighCardinalityKeyValues(TaxContext context) {
return KeyValues.of(USER_ID.withValue(context.getUserId()));
}
@Override
public String getName() {
return "default.tax.name";
}
}
/**
* If micrometer-docs-generator is used, we will automatically generate documentation
* for your observations. Check this URL
* https://github.com/micrometer-metrics/micrometer-docs-generator#documentation for
* setup example and read the {@link ObservationDocumentation} javadocs.
*/
enum TaxObservationDocumentation implements ObservationDocumentation {
CALCULATE {
@Override
public Class<? extends ObservationConvention<? extends Observation.Context>> getDefaultConvention() {
return DefaultTaxObservationConvention.class;
}
@Override
public String getContextualName() {
return "calculate tax";
}
@Override
public String getPrefix() {
return "tax";
}
@Override
public KeyName[] getLowCardinalityKeyNames() {
return TaxLowCardinalityKeyNames.values();
}
@Override
public KeyName[] getHighCardinalityKeyNames() {
return TaxHighCardinalityKeyNames.values();
}
};
enum TaxLowCardinalityKeyNames implements KeyName {
TAX_TYPE {
@Override
public String asString() {
return "tax.type";
}
}
}
enum TaxHighCardinalityKeyNames implements KeyName {
USER_ID {
@Override
public String asString() {
return "tax.user.id";
}
}
}
}
/**
* Our business logic that we want to observe.
*/
class TaxCalculator {
private final ObservationRegistry observationRegistry;
// If the user wants to override the default they can override this. Otherwise,
// it will be {@code null}.
@Nullable
private final TaxObservationConvention observationConvention;
TaxCalculator(ObservationRegistry observationRegistry,
@Nullable TaxObservationConvention observationConvention) {
this.observationRegistry = observationRegistry;
this.observationConvention = observationConvention;
}
void calculateTax(String taxType, String userId) {
// Create a new context
TaxContext taxContext = new TaxContext(taxType, userId);
// Create a new observation
TaxObservationDocumentation.CALCULATE
.observation(this.observationConvention, new DefaultTaxObservationConvention(), () -> taxContext,
this.observationRegistry)
// Run the actual logic you want to observe
.observe(this::calculateInterest);
}
private void calculateInterest() {
// do some work
}
}
/**
* Example of user changing the default conventions.
*/
class CustomTaxObservationConvention extends DefaultTaxObservationConvention {
@Override
public KeyValues getLowCardinalityKeyValues(TaxContext context) {
return super.getLowCardinalityKeyValues(context)
.and(KeyValue.of("additional.low.cardinality.tag", "value"));
}
@Override
public KeyValues getHighCardinalityKeyValues(TaxContext context) {
return KeyValues.of("this.would.override.the.default.high.cardinality.tags", "value");
}
@Override
public String getName() {
return "tax.calculate";
}
}
/**
* A utility class to set cloud related arguments.
*/
static class CloudUtils {
static String getZone() {
return "...";
}
static String getCloudInstanceId() {
return "...";
}
}For a more detailed example, see the full usage example of an instrumentation together with overriding the default tags.
The following example puts the whole code together:
// Registry setup
ObservationRegistry observationRegistry = ObservationRegistry.create();
// add metrics
SimpleMeterRegistry registry = new SimpleMeterRegistry();
observationRegistry.observationConfig().observationHandler(new DefaultMeterObservationHandler(registry));
observationRegistry.observationConfig().observationConvention(new GlobalTaxObservationConvention());
// This will be applied to all observations
observationRegistry.observationConfig().observationFilter(new CloudObservationFilter());
// In this case we're overriding the default convention by passing the custom one
TaxCalculator taxCalculator = new TaxCalculator(observationRegistry, new CustomTaxObservationConvention());
// run the logic you want to observe
taxCalculator.calculateTax("INCOME_TAX", "1234567890");Observation Predicates and Filters
To globally disable observations under given conditions, you can use an ObservationPredicate. To mutate the Observation.Context, you can use an ObservationFilter.
To set these, call the ObservationRegistry#observationConfig()#observationPredicate() and ObservationRegistry#observationConfig()#observationFilter() methods, respectively.
The following example uses predicates and filters:
// Example using a metrics handler - we need a MeterRegistry
MeterRegistry meterRegistry = new SimpleMeterRegistry();
// Create an ObservationRegistry
ObservationRegistry registry = ObservationRegistry.create();
// Add predicates and filter to the registry
registry.observationConfig()
// ObservationPredicate can decide whether an observation should be
// ignored or not
.observationPredicate((observationName, context) -> {
// Creates a noop observation if observation name is of given name
if ("to.ignore".equals(observationName)) {
// Will be ignored
return false;
}
if (context instanceof MyContext) {
// For the custom context will ignore a user with a given name
return !"user to ignore".equals(((MyContext) context).getUsername());
}
// Will proceed for all other types of context
return true;
})
// ObservationFilter can modify a context
.observationFilter(context -> {
// We're adding a low cardinality key to all contexts
context.addLowCardinalityKeyValue(KeyValue.of("low.cardinality.key", "low cardinality value"));
if (context instanceof MyContext) {
// We're mutating a specific type of a context
MyContext myContext = (MyContext) context;
myContext.setUsername("some username");
// We want to remove a high cardinality key value
return myContext.removeHighCardinalityKeyValue("high.cardinality.key.to.ignore");
}
return context;
})
// Example of using metrics
.observationHandler(new DefaultMeterObservationHandler(meterRegistry));
// Observation will be ignored because of the name
then(Observation.start("to.ignore", () -> new MyContext("don't ignore"), registry)).isSameAs(Observation.NOOP);
// Observation will be ignored because of the entries in MyContext
then(Observation.start("not.to.ignore", () -> new MyContext("user to ignore"), registry))
.isSameAs(Observation.NOOP);
// Observation will not be ignored...
MyContext myContext = new MyContext("user not to ignore");
myContext.addHighCardinalityKeyValue(KeyValue.of("high.cardinality.key.to.ignore", "some value"));
Observation.createNotStarted("not.to.ignore", () -> myContext, registry).observe(this::yourCodeToMeasure);
// ...and will have the context mutated
then(myContext.getLowCardinalityKeyValue("low.cardinality.key").getValue()).isEqualTo("low cardinality value");
then(myContext.getUsername()).isEqualTo("some username");
then(myContext.getHighCardinalityKeyValues())
.doesNotContain(KeyValue.of("high.cardinality.key.to.ignore", "some value"));Using Annotations With @Observed
If you have turned on Aspect Oriented Programming (for example, by using org.aspectj:aspectjweaver), you can use the @Observed annotation to create observations. You can put that annotation either on a method to observe it or on a class to observe all the methods in it.
The following example shows an ObservedService that has an annotation on a method:
static class ObservedService {
@Observed(name = "test.call", contextualName = "test#call",
lowCardinalityKeyValues = { "abc", "123", "test", "42" })
void call() {
System.out.println("call");
}
}The following test asserts whether the proper observation gets created when a proxied ObservedService instance gets called:
// create a test registry
TestObservationRegistry registry = TestObservationRegistry.create();
// add a system out printing handler
registry.observationConfig().observationHandler(new ObservationTextPublisher());
// create a proxy around the observed service
AspectJProxyFactory pf = new AspectJProxyFactory(new ObservedService());
pf.addAspect(new ObservedAspect(registry));
// make a call
ObservedService service = pf.getProxy();
service.call();
// assert that observation has been properly created
TestObservationRegistryAssert.assertThat(registry)
.hasSingleObservationThat()
.hasBeenStopped()
.hasNameEqualTo("test.call")
.hasContextualNameEqualTo("test#call")
.hasLowCardinalityKeyValue("abc", "123")
.hasLowCardinalityKeyValue("test", "42")
.hasLowCardinalityKeyValue("class", ObservedService.class.getName())
.hasLowCardinalityKeyValue("method", "call").doesNotHaveError();
