Spring Boot Observability: OpenTelemetry and Grafana Setup

"We have logs" means this when a production problem hits: someone connects to the server, searches the file by time range and tries to match the logs of three services by hand. Observability is the opposite: answering "why did this request take 4 seconds" within minutes, on a single screen that shows which service waited how long on which query.

Spring Boot 3 makes this possible with almost no code: Micrometer metrics, OpenTelemetry traces through Micrometer Tracing and structured logs carrying the traceId. This article covers what the three signals are for, the setup steps, adding your own business metrics, Grafana dashboards and SLO-based alerts, and the cost and cardinality pitfalls we see most often.

Three signals, three questions

What ties the three together is the traceId: in Grafana you go from a slow-request graph to the trace, and from the trace to that request's logs in one click. Without correlation, it's three separate searches in three separate tools.

Setup: dependencies and configuration

<dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-actuator</artifactId></dependency>
<dependency><groupId>io.micrometer</groupId><artifactId>micrometer-registry-prometheus</artifactId></dependency>
<dependency><groupId>io.micrometer</groupId><artifactId>micrometer-tracing-bridge-otel</artifactId></dependency>
<dependency><groupId>io.opentelemetry</groupId><artifactId>opentelemetry-exporter-otlp</artifactId></dependency>
<dependency><groupId>net.ttddyy.observation</groupId><artifactId>datasource-micrometer-spring-boot</artifactId></dependency>  <!-- JDBC queries in traces -->

management:
  endpoints:
    web:
      exposure:
        include: health, prometheus, info      # actuator endpoints; must not be public
  tracing:
    sampling:
      probability: 1.0                          # 0.1–0.2 in production; tail sampling for error traces
  otlp:
    tracing:
      endpoint: http://otel-collector:4318/v1/traces
  metrics:
    tags:
      application: order-service                # an application tag on every metric
spring:
  application:
    name: order-service
  kafka:
    template:
      observation-enabled: true                 # Kafka producer/consumer traces
    listener:
      observation-enabled: true
logging:
  structured:
    format:
      console: ecs                              # Spring Boot 3.4+: JSON logs including traceId/spanId

With just this, and no code, you get: request count, error rate and latency histogram for every HTTP endpoint; spans for RestClient/WebClient calls, JDBC queries and Kafka messages; JVM memory, GC, thread and HikariCP pool metrics; JSON logs with traceId and spanId.

Adding your own observations

Infrastructure metrics answer "is the server healthy"; "is the business healthy" needs business metrics. Two tools are enough:

@Service
public class OrderService {
    private final Counter placed;
    private final Timer approvalTime;

    public OrderService(MeterRegistry registry) {
        this.placed = Counter.builder("orders.placed")
            .description("Number of orders placed").register(registry);
        this.approvalTime = Timer.builder("orders.approval.duration")
            .publishPercentileHistogram().register(registry);
    }

    @Observed(name = "orders.place", contextualName = "place-order")   // metric + span together
    public OrderNumber place(OrderRequest request, Channel channel) {
        OrderNumber number = ...;
        placed.increment();
        registry.counter("orders.placed.by.channel", "channel", channel.name()).increment();   // low-cardinality tag
        return number;
    }
}

@Observed needs an ObservedAspect bean; the annotation produces a Timer and a span at the same time. The tag rule is critical: tag values must be a bounded set (channel, status, region). A user id, an order number or an e-mail can't be a tag; every distinct value opens a new time series and brings Prometheus down. That kind of information goes into the trace (span attribute) or the log, not the metric.

Dashboards: RED and USE

A good dashboard answers questions, it doesn't exhibit graphs. Two standard methods are enough:

• RED (for every service and endpoint): Rate — requests per second; Errors — 5xx ratio; Duration — p50/p95/p99 latency.

• USE (for every resource): Utilization — CPU, memory, connection pool usage; Saturation — queues and waits; Errors — pool timeouts, GC pressure.

# p99 latency per endpoint (Prometheus, from the histogram)
histogram_quantile(0.99,
  sum by (le, uri) (rate(http_server_requests_seconds_bucket{application="order-service"}[5m])))

# Error rate
sum(rate(http_server_requests_seconds_count{status=~"5.."}[5m]))
  / sum(rate(http_server_requests_seconds_count[5m]))

# HikariCP: requests waiting for a connection
hikaricp_connections_pending{application="order-service"}

Alerts: look at the user, not the CPU

A "CPU 80%" alert wakes you at three in the morning but says nothing about whether users are affected. Alerts are built on SLOs (service level objectives): a target such as "99.5% of requests succeed in under 500 ms", an error budget (the acceptable share of failed requests per month) and alerts tied to the budget's burn rate. If 5% of the budget burns in two hours, page; if in two days, open a ticket. Resource metrics (CPU, pools) are a diagnosis dashboard, not an alert.

Log correlation and discipline

• traceId on every log line: Automatic with structured logging on; to carry MDC across your own threads you need micrometer-context-propagation.

• Event logs, not narrative logs: Instead of "Creating order…", "Order created", one line: event, order number, duration, outcome.

• No personal data: A log filter masks phone numbers, e-mails and identity numbers.

• Level discipline: ERROR only for situations someone must look at; a business-rule rejection (out of stock) is not even WARN, it's INFO.

Cost: cardinality, sampling, volume

The cost of observability infrastructure grows with three things. Cardinality: high-variety tags explode metric storage; keep tag values in the hundreds at most. Trace sampling: tracing every request is expensive; head sampling (10–20%) is cheap but can miss failed requests, and tail sampling in the OpenTelemetry Collector ("always keep error and slow traces, 5% of the rest") gives the best balance. Log volume: DEBUG off in production, health-check requests not logged, retention by signal (metrics 13 months, traces 7–14 days, logs 30 days, for example).

Pitfalls

1. Exposing actuator to the internet: /actuator/prometheus and /actuator/env only from the internal network or with authentication; a separate rule in Spring Security.

2. Losing context at async boundaries: The traceId doesn't cross @Async, CompletableFuture and virtual threads on its own — it is carried with the context-propagation library and ContextExecutorService.

3. Dropping headers in Kafka: Trace context travels in message headers; if hand-built messages don't copy the headers, the trace breaks on the consumer side.

4. Not measuring timeouts: The duration metric for external calls must include timeout errors; otherwise the slowest requests never appear on the graph.

5. Dashboards nobody looks at: Dashboards should be part of the post-release checklist and the weekly operations meeting.

Frequently asked questions

OpenTelemetry Java agent or Micrometer Tracing?

In Spring Boot 3 projects, Micrometer Tracing plus the OTel bridge is our recommendation because it integrates with Spring's own observation points (the Observation API). In legacy applications where code can't be changed, the OTel Java agent (bytecode instrumentation) is the only option; the two are not used together.

Is this infrastructure too much for a small team?

Grafana, Prometheus, Tempo and Loki start with a single Docker Compose file, or you use managed Grafana Cloud. Setup is a day's work; it pays for itself at the first production incident.

Why are traces needed in a modular monolith?

Even in a single deployment unit, a request passes through five modules and ten queries; the trace shows which module spent the time. Spring Modulith's observability module turns module entries into spans automatically.

Conclusion

Observability in Spring Boot 3 starts with a few dependencies and a configuration file; its value shows in discipline: low-cardinality business metrics, logs tied together by traceId, RED/USE dashboards and SLO-based alerts. These four habits make production problems solvable in minutes instead of hours.

To set up the observability stack for your production application and move to SLO-based monitoring, see our maintenance and support service.