Skip to content
Menu
All work

Practice Reliability engineering

Reliability patterns for enterprise integration

Enterprise integration is mostly not about the happy path; the happy path takes an afternoon. These are the four patterns I reach for, drawn as they actually behave, with the cost of each stated rather than implied. Every one of them appears in both case studies.

04

The patterns

Retry with backoff

01

A transient fault: the endpoint was briefly unreachable, throttled, or slow enough to time out. The same request would succeed a moment later.

A caller retries a failing service three times, with the interval growing between each attempt, up to a fixed attempt limit. attempt 1 · immediate attempt 2 · +2s attempt 3 · +4s, then stop Caller idempotent op Service transient fault add jitter, or every caller retries on the same schedule
Three attempts, each further apart than the last, then stop.
When to reach for it
When the operation is idempotent and the fault is genuinely transient. Bounded, always — an attempt limit and a growing interval.
What it costs
Latency amplification under load, and a retry storm if every caller backs off on the same schedule. Add jitter, or the retries synchronise and become the outage.

Dead-lettering

02

A message that will never succeed: malformed payload, a contract that changed, a reference to something that no longer exists. Retrying it forever is just load.

A queue feeds a handler that retries within a budget; once that budget is spent the message is routed to a dead-letter queue, which raises an alert to a named owner. budget spent alert bounded retry Queue Handler n attempts Dead letter Owner runbook
The retry budget ends somewhere, and that somewhere needs an owner.
When to reach for it
Immediately after the retry budget is exhausted. The queue needs a named owner, an alert and a runbook before it is switched on.
What it costs
An operational obligation. A dead-letter queue nobody watches is strictly worse than no dead-letter queue, because it looks like handling.

Circuit breaking

03

A dependency that is down rather than slow. Every caller is queueing against it, holding connections, and the retries are making the recovery slower.

A circuit breaker moves from closed to open once a failure threshold is crossed, to half-open after a cool-down, and then back to closed if a probe succeeds or to open if it fails. failure threshold cool-down elapsed probe fails probe succeeds Closed calls pass Open fail fast Half-open one probe
Three states. The half-open probe is the part people leave out.
When to reach for it
In front of a downstream that can be overwhelmed, especially one shared by several callers. Needs a half-open probe to discover recovery.
What it costs
It fails requests that might have succeeded, and it moves the failure to the caller, who now needs a fallback. Tuning the thresholds is genuinely hard.

Idempotent processing

04

At-least-once delivery doing exactly what it promises: the same message arriving twice, or a third party replaying yesterday because their side had an incident.

A message and its redelivery both carry the same business key. The handler applies the effect for the first and discards the second, so the effect happens exactly once. Message key K Redelivery key K Handler seen(K)? Effect applied once Discarded no second effect
Two deliveries, one business key, one effect.
When to reach for it
Always, in a messaging estate. Key on a business identifier, not on a delivery or message id, which changes on redelivery.
What it costs
Every handler carries it, and there is no central enforcement — it lives or dies on code review. Storage for the seen-keys record, and a decision about how long to keep it.

Where these show up

All four are load-bearing in the integration platform and in the estate rescue, where redesigning retry and dead-lettering together is what moved the incident rate. The longer argument is in Designing for the third party having a bad day.

All work