Webhook Design: Signatures, Retries, Ordering, Idempotency

A webhook is the reverse of an API: you don't ask for the data; when something happens, the other side calls your endpoint. The payment provider says "payment completed", the carrier says "delivered", the e-invoice integrator says "invoice rejected". It looks simple; in the field, network errors, timeouts, the same event arriving three times, events arriving out of order and forged requests complicate it fast. A badly designed webhook integration either misses events or processes them twice.

This article covers both sides: if your product sends webhooks (provider) — the event envelope, HMAC signatures, delivery from an outbox, the retry schedule and ordering; if you receive webhooks (consumer) — signature verification, a fast 200, asynchronous and idempotent processing. At the end, a security checklist and common mistakes.

Webhook or polling?

Criterion	Webhook	Polling (periodic query)
Latency	Seconds	The polling interval (minutes)
Load	Only when an event occurs	A request every interval, event or not
Reliability	Delivery guarantee depends on the provider's retries	Simple; a missed event is caught on the next query
Complexity	Public endpoint, signatures, idempotency	Client logic only, a "last checked" cursor

The most robust production setup combines the two: the webhook is the main path, a low-frequency reconciliation query (for example, "events of the last hour" once an hour) is the backup. A missed webhook is caught in reconciliation.

The provider side: sending webhooks

The event envelope

{
  "id": "evt_01J9X3ZK7Q8W2M",             // unique event id → consumer idempotency
  "type": "order.shipped",                 // resource.event form, versioned catalog
  "version": 2,                            // schema version
  "occurred_at": "2026-11-17T09:41:05Z",   // event time (not delivery time)
  "sequence": 8123,                        // increasing sequence per resource
  "data": { "order_number": "S-2026-004512", "tracking": "TR123456789" }
}

Two design decisions: thin or fat? A fat envelope (all data inside) saves the consumer a call; a thin envelope (ids only, the consumer fetches from the API) keeps sensitive data off the webhook and always yields current state. For events carrying payment or personal data, thin is preferred.

Delivery from an outbox

A webhook is not sent with an HTTP call from inside the business transaction: if the consumer is slow your transaction waits, if the consumer is down the event is lost. The event is written to an outbox table in the same transaction as the business data; a separate delivery worker sends to subscribers, records the outcome and retries on failure.

HMAC signatures

public String sign(String timestamp, byte[] body, byte[] secret) throws GeneralSecurityException {
    Mac mac = Mac.getInstance("HmacSHA256");
    mac.init(new SecretKeySpec(secret, "HmacSHA256"));
    mac.update(timestamp.getBytes(StandardCharsets.UTF_8));
    mac.update((byte) '.');
    mac.update(body);
    return HexFormat.of().formatHex(mac.doFinal());
}
// Headers:
// Webhook-Id: evt_01J9X3ZK7Q8W2M
// Webhook-Timestamp: 1794000065
// Webhook-Signature: v1=3f2a…   (several signatures allowed: v1=old,v1=new during key rotation)

The signature is computed over the raw bytes of the body; the timestamp is part of the signature so a captured request can't be replayed days later. The secret is per subscriber and rotatable from the dashboard; during rotation, requests are signed with both keys.

Retry schedule and failure handling

Only 2xx is success; 3xx isn't followed, 4xx (401/403/404/410) is a configuration error that produces an alert instead of retries, 5xx and timeouts go into retry.
Keep the timeout short (5–10 s); the consumer must do its work asynchronously.
Exponential schedule: 1 min, 5 min, 30 min, 2 h, 12 h, 24 h, 48 h — about 3 days in total; jitter on every attempt.
When the schedule is exhausted, the event lands on an "undelivered" list; the subscriber sees it in the dashboard and can resend manually. If an endpoint fails for days, the subscription is suspended automatically and the subscriber is notified by e-mail.
Per-subscriber concurrency limit and circuit breaker: one slow subscriber doesn't delay the others' deliveries.

Ordering: don't guarantee it, make it manageable

Because of retries and parallel delivery, events may not arrive in order; "shipped" can arrive before "preparing". Guaranteeing order slows delivery badly. Instead, put sequence and occurred_at in the envelope; the consumer skips an event older than the resource's last known sequence, or fetches current state from the API when needed. Scenarios that need strict ordering are queue work, not webhook work.

The consumer side: receiving webhooks

@PostMapping("/webhooks/payment")
public ResponseEntity<Void> paymentWebhook(@RequestHeader("Webhook-Id") String eventId,
                                            @RequestHeader("Webhook-Timestamp") long timestamp,
                                            @RequestHeader("Webhook-Signature") String signature,
                                            @RequestBody byte[] rawBody) {
    if (Math.abs(Instant.now().getEpochSecond() - timestamp) > 300) return ResponseEntity.status(400).build();   // 5-minute tolerance
    if (!verifier.valid(timestamp, rawBody, signature)) return ResponseEntity.status(401).build();               // constant-time comparison
    boolean isNew = inboundEvents.saveIfAbsent(eventId, rawBody);                                                 // unique constraint: a repeated event is skipped
    if (isNew) queue.send(new ProcessWebhook(eventId));                                                          // processing is asynchronous
    return ResponseEntity.ok().build();                                                                          // fast 200: the provider doesn't wait
}

Four rules: verify over the raw body (text re-serialized from JSON breaks the signature); store first, return 200 immediately, process afterwards; be idempotent by event id (if the same event arrives again it's already stored and skipped); a processing failure doesn't affect the webhook response — the failure is retried in your own queue. During processing, check the event's order: if it's below the last sequence on record, it's stale.

Security checklist

HTTPS mandatory; no endpoint without signature verification.
Replay protection through timestamp tolerance and an event-id table.
Secrets per subscriber, rotatable; in secret management, not in code.
Endpoint URLs unguessable and tied to identity; optional IP allow-list (if the provider publishes one).
Body size and rate limits; the webhook endpoint must not consume the same pool as the main API.
Raw body and headers logged (personal data masked); "what arrived" can be answered when something goes wrong.

Testing and operations

As a provider, give subscribers a "send test event" button and a delivery history (request, response code, duration); most integration problems are solved on that screen. As a consumer, use the provider's sandbox and a tunnel tool for local development; keep recorded real events as test data and replay them in integration tests.

Common mistakes

Sending the webhook from inside the business transaction — the event is lost when the consumer is down, and the transaction waits.
Processing synchronously on the consumer side — 30 seconds of work hits the provider's timeout, a retry arrives, the work is done twice.
No idempotency by event id — retries create duplicate records.
Computing the signature over a re-serialized body — the signature breaks when field order changes.
Assuming ordering — a "preparing" event arriving after "cancelled" reopens the order.
No reconciliation — when the three-day retry window expires, the event is silently gone.

Frequently asked questions

Should we use Kafka or a queue instead of webhooks?

Between systems inside the same organization, a queue is more reliable. Webhooks are the standard way to deliver events to external customers and partners — parties that have no access to your queue.

How long should we keep events if the consumer doesn't respond?

Active for the retry window (3 days); afterwards, undelivered events stay resendable by hand for 30 days. With an event-catalog API, the consumer should also be able to query past events.

How do we deliver one event to several subscribers?

The event in the outbox is fanned out into one delivery record per subscriber according to the subscription table; each delivery has its own status, attempts and signature.

Conclusion

Good webhook design is the work of both sides: the provider guarantees "delivering" with the event envelope, an outbox, HMAC signatures and a retry schedule; the consumer guarantees "processing" with signature verification, a fast 200 and idempotent asynchronous processing keyed by event id. Ordering isn't guaranteed but is managed through the sequence in the envelope; missed events are caught by reconciliation.

To add webhook infrastructure to your product or make your existing webhook integrations reliable, see our integration and automation service.