Circuit Breaker, Retry and Bulkhead with Resilience4j

The most destructive kind of failure in production isn't the failure itself but its spread: the ERP's stock service slows down, every request waits 30 seconds, Tomcat's 200 threads fill with waiting requests, and even the login screen — which has nothing to do with stock — stops opening. This is a cascading failure, and the fix isn't making the external service faster; it's cutting the dependency on it in a controlled way.

Resilience4j is the standard toolkit for that control in Java. This article covers its five modules — circuit breaker, retry, time limiter, bulkhead and rate limiter — when each is needed, the Spring Boot setup, configuration values that work in production, fallback strategies, measurement and the mistakes we see most often in the field.

Five modules, five different questions

The circuit breaker: three states

CLOSED (normal): calls go through, outcomes are counted in a sliding window. When the failure rate or slow-call rate crosses the threshold, OPEN: calls are rejected immediately with CallNotPermittedException without being attempted; no thread waits, the fallback kicks in. When the wait duration expires, HALF_OPEN: a limited number of trial calls go through; if they succeed the breaker closes, otherwise it opens again. The critical point: an open circuit is not an error, it's a protection mechanism — the user sees "last known stock" instead of a blank screen, and the system stays up.

Spring Boot setup

<dependency><groupId>io.github.resilience4j</groupId><artifactId>resilience4j-spring-boot3</artifactId></dependency>
<dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-aop</artifactId></dependency>
<dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-actuator</artifactId></dependency>

resilience4j:
  circuitbreaker:
    instances:
      sap:
        slidingWindowType: COUNT_BASED
        slidingWindowSize: 50                 # the last 50 calls
        minimumNumberOfCalls: 20              # at least 20 calls before deciding
        failureRateThreshold: 50              # 50% failures → open
        slowCallDurationThreshold: 2s         # ~2× the p99
        slowCallRateThreshold: 60             # 60% slow → open
        waitDurationInOpenState: 30s
        permittedNumberOfCallsInHalfOpenState: 5
        ignoreExceptions:
          - com.acme.erp.sap.SapBusinessRuleException  # 4xx-like business errors don't count as failures
  retry:
    instances:
      sap:
        maxAttempts: 3
        waitDuration: 500ms
        enableExponentialBackoff: true
        exponentialBackoffMultiplier: 2
        enableRandomizedWait: true            # jitter: don't let everyone retry at the same instant
        retryExceptions:
          - java.io.IOException
          - org.springframework.web.client.HttpServerErrorException
        ignoreExceptions:
          - org.springframework.web.client.HttpClientErrorException
  timelimiter:
    instances:
      sap:
        timeoutDuration: 3s
        cancelRunningFuture: true
  bulkhead:
    instances:
      sap:
        maxConcurrentCalls: 25
        maxWaitDuration: 100ms

@Component
public class SapStockAdapter implements StockSource {
    private final SapOdataClient client;
    private final StockCache cache;

    @Override
    @CircuitBreaker(name = "sap", fallbackMethod = "lastKnown")
    @Retry(name = "sap")
    @Bulkhead(name = "sap")
    @TimeLimiter(name = "sap")
    public CompletableFuture<StockStatus> query(ProductCode product, WarehouseCode warehouse) {
        return CompletableFuture.supplyAsync(() -> client.stock(product, warehouse), executor)
                                .thenApply(SapStockMapping::toStockStatus);
    }

    // fallback: same return type, last parameter Throwable (or a specific exception)
    CompletableFuture<StockStatus> lastKnown(ProductCode product, WarehouseCode warehouse, Throwable t) {
        return CompletableFuture.completedFuture(
            cache.lastValue(product, warehouse).orElse(StockStatus.unknown(product, warehouse)));
    }
}

By default the annotations apply outside-in as Retry → CircuitBreaker → RateLimiter → TimeLimiter → Bulkhead: a timeout or a bulkhead rejection counts as a failure for the breaker, and if the breaker is open, retry tries again (retrying an open circuit is pointless; add CallNotPermittedException to retry's ignore list). @TimeLimiter only works on methods returning CompletableFuture; with synchronous clients, the timeout is set on the HTTP client itself (connect and read timeouts).

Fallback: design what you'll return up front

A circuit breaker without a fallback only changes the type of the error. There are four strategies, and the choice is a business decision:

• Last known value: For reads such as stock, price or exchange rate, the value "from 12 minutes ago" from a cache, shown with a "may be outdated" marker.

• Default / degraded mode: Popular products if the recommendation service is down; "route to approval" if the credit score is unavailable.

• Deferred operation: For writes, put the request in an outbox or queue, say "accepted for processing", send when the other side is back.

• Fast, explicit failure: For operations with no substitute, such as payment, a clear message to the user rather than a technical error.

What not to do: a fallback that calls the same dependency by another route (the protection disappears), or one that swallows the error silently and returns an empty list (nobody notices).

How to pick configuration values

• Timeout budget: If the user request's total timeout is 10 s, the inner external call gets 3 s, and with retries the total can't exceed 3 attempts × 3 s; the inner timeout must always be shorter than the outer one.

• Slow-call threshold: About twice the dependency's normal p99; writing "2 seconds" without measuring opens either too early or too late.

• Window and minimum calls: For a low-traffic dependency a time-based window (TIME_BASED, 60 s) is more stable; minimumNumberOfCalls stops a single error from opening the circuit.

• Retry count: 3 attempts, 500 ms initial wait, 2× multiplier, jitter on. More than that creates a "retry storm" on the other system.

• Bulkhead: The dependency's reasonable concurrency (for example 25 for the ERP) plus a short wait; with virtual threads a semaphore-based bulkhead is enough, a thread-pool bulkhead is unnecessary.

Measurement and alerts

Resilience4j feeds Micrometer automatically: resilience4j.circuitbreaker.state (which circuit is open), resilience4j.circuitbreaker.calls (successful/failed/slow), resilience4j.retry.calls (succeeded on which attempt), resilience4j.bulkhead.available.concurrent.calls. /actuator/circuitbreakers and /actuator/circuitbreakerevents show the live state. The alert rule is simple: notify when a circuit stays open for more than 5 minutes; if the retry rate is persistently high (20% of requests succeed on the second attempt), it's time to talk to the dependency's owners.

Common mistakes in the field

1. Retrying a non-idempotent POST. Retrying order creation means duplicate orders; retry only idempotent calls, or use an idempotency key.

2. Nested retries. 3 in the HTTP client, 3 in Resilience4j, 3 in the calling service: 27 requests. Retry belongs in one layer.

3. One circuit breaker for all dependencies. When the ERP's stock endpoint breaks, the price endpoint closes too; name breakers per dependency and operation.

4. Counting business errors as failures. "Product not found" (404) must not open the circuit; separate it with ignoreExceptions.

5. A circuit breaker without a timeout. The breaker can't open without seeing failures; calls waiting 30 seconds count as "success", not failure. Timeout first, breaker second.

6. An untested fallback. The fallback runs for the first time in production and throws NullPointerException; test the fallback path in an integration test by deliberately taking the dependency down.

Frequently asked questions

Spring Cloud Circuit Breaker or Resilience4j directly?

Spring Cloud Circuit Breaker is an abstraction on top of Resilience4j, designed for swapping providers. In most projects with a single provider, the Resilience4j annotations directly give less abstraction and more tuning options.

Are circuit breakers used in message consumers too?

Yes; if the Kafka consumer can't write to the ERP the circuit opens and the consumer should pause processing and wait; instead of filling a dead-letter queue, it continues where it left off once the circuit closes.

Are these patterns needed outside microservices?

Even a modular monolith has external dependencies such as the ERP, payments, e-invoicing and e-mail; every external call is a cascading-failure candidate. The patterns depend on external dependencies, not on the architecture.

Conclusion

Resilience doesn't mean external services never fail; it means you decided in advance what your system does when they do: a timeout budget, measured retries only for idempotent work, a circuit breaker per dependency, a designed fallback and an open-circuit alert. Resilience4j makes those decisions implementable with a few annotations and a YAML file.

To make your existing integrations resilient against cascading failure, see our integration and automation and maintenance and support services.