Designing for the third party having a bad day

Enterprise integration is mostly not about the happy path. The happy path takes an afternoon. What takes seven years to learn is what the system does when the thing on the other end is slow, returns a 500, accepts a message and loses it, or comes back and asks for everything again.

I took over an integration estate once where none of this had been designed. Undocumented interfaces, deployments done by hand through the portal, and failures that recurred without anyone knowing why. Bringing it under control cut recurring production incidents by about 30% — and almost none of that came from new features. It came from the patterns below.

Retry, and knowing what not to retry

The default instinct is to retry everything. That’s actively harmful.

A 500 or a timeout is worth retrying — the operation may succeed later. A 400 or a 422 is not: the message is malformed and it will be malformed on every subsequent attempt. Retrying it burns throughput, delays the queue behind it, and fills your logs with noise that hides the failures you could have fixed.

So retries need to distinguish transient from permanent, and use exponential backoff with jitter. Fixed-interval retry across a fleet produces a thundering herd that arrives in lockstep precisely when the downstream system is least able to cope — you’ve built a system that synchronises its own attacks on a service that’s already struggling.

Dead-letter as a designed destination

Every message that can’t be processed has to go somewhere deliberate. The dead-letter queue is where the system admits defeat in a way a human can act on.

What makes one useful rather than a graveyard:

• The original message, unmodified. If you can’t replay it exactly, you can’t recover.
• The failure reason and the attempt count, so you know whether it exhausted retries or was rejected outright.
• An alert when depth crosses a threshold. An unmonitored dead-letter queue is just data loss with extra steps, and it is remarkable how long one can sit there quietly accumulating.
• A replay path that’s been rehearsed. The worst time to discover your replay doesn’t work is during an incident.

Idempotency

Any system with retries will eventually deliver the same message twice. That isn’t an edge case — it’s a guarantee, and at-least-once delivery means the design has to assume it.

The consumer needs a stable idempotency key and a record of what it has already processed, so a duplicate is recognised and discarded rather than producing a second order, a second payment, or a second ticket. If the receiving system can’t be made idempotent, the integration layer has to deduplicate before it gets there.

This is the pattern most often skipped, because in testing you rarely see the duplicate. In production it arrives eventually, and it usually arrives as a finance query.

Circuit breaking

When a downstream system is failing, continuing to send is worse than stopping. You add load to something that’s already unhealthy, and you convert a fast failure into a slow one — each call waiting for a timeout, holding a connection, backing up the queue behind it.

A circuit breaker fails fast once a threshold is crossed, then probes periodically to see if recovery has happened. The value isn’t only protecting the downstream: it’s that your own system stays responsive, and the failure is visible immediately rather than as gradually increasing latency nobody notices until it’s an outage.

Instrumenting the failure path

The pattern underneath all of these: the failure path deserves the same attention as the success path, and usually gets none.

Most integration estates are well instrumented for throughput and completely blind to how they fail. What proved most valuable was alerting on failure paths directly — dead-letter depth, retry exhaustion rate, circuit-breaker state — so problems announced themselves instead of being discovered by a business user asking why their report was empty.

That shift, from finding out downstream to being told upstream, is most of what “improving stability” actually means in practice. The interfaces didn’t get better. We just stopped being the last to know.

Patterns applied across Azure Logic Apps, Service Bus, Event Grid, Azure Functions, and API Management.