02 Estate turnaround Automotive 2022 — 2024

Taking over an estate nobody had documented

I inherited the integration estate of a global 4x4 vehicle manufacturer. Nobody could say with confidence what interfaces existed, deployments were done by hand in the portal, and there was a standing population of failures that recurred often enough to be normal and were never explained. This is what I did first, and why that order.

~30%: fewer recurring incidents
~15%: lower Azure consumption cost
30+: interfaces brought under CI/CD

Context

The estate worked, mostly. That is what makes this kind of situation hard to argue for fixing: there was no outage to point at, just a persistent background rate of things going wrong that the team had learned to absorb. Support knew which interfaces to re-run manually in the morning. That knowledge lived in people, not documents.

Meanwhile the business kept asking for new interfaces, and there was no version of this where delivery stopped while the platform got fixed. Everything below had to happen alongside a live delivery commitment.

Constraints

No stopping for repairs: New interface delivery continued throughout; none of this was a funded remediation programme.
Unknown inventory: No reliable record of what existed, what depended on it, or which interfaces were still in use.
Failures without explanation: Recurring, absorbed by manual workarounds, and invisible above the level of individual interfaces.
Drifting cost: Consumption spend rising without a corresponding rise in what the estate did.

Architecture

1 Before: a change begins on a developer machine with no reviewable record.
2 Before: it reaches production through the portal, by hand, with no artefact to roll back to.
3 After: the repository is the record — reviewed, versioned, and the only source of a deployment.
4 After: one pipeline, Bicep and YAML, producing a single artefact.
5 After: that same artefact promoted through three environments rather than rebuilt per environment.
6 After: every failure path raises an alert that reaches a named owner — the change that moved the incident numbers.

The release and failure path, before and after. The second row is the whole intervention.

Decisions

What was considered, what was chosen, and what that choice cost. The last line is the one that matters.

Decision 01

Instrument before repairing anything

Considered: rewrite the interfaces that failed most · document the estate first, then prioritise · instrument every failure path first
Chose: Proactive alerting on every failure path before touching a single interface design.
Because: "Recurring unexplained failures" is an observability problem before it is a design problem. Rewriting the interfaces that appeared to fail most would have optimised against the sample that happened to be visible, which is not the same as the sample that mattered. Documentation first was the respectable-sounding option and it would have taken months to produce something already out of date.
Gave up: Several months in which the estate looked significantly worse. The alerts surfaced failures that had always been happening silently, and I had to keep explaining that the number going up was the instrument working, not the platform degrading.

Decision 02

Automate deployment before improving design

Considered: fix the interfaces, then automate releases · automate releases, then fix the interfaces
Chose: Azure DevOps pipelines across three environments, first.
Because: Every design fix has to be deployed, and manual portal deployment made each fix carry its own risk of introducing a new fault. Automating first turned the release step from a hazard into a non-event, which made every subsequent improvement cheaper and safer. It is a multiplier, and multipliers go first.
Gave up: The first visible deliverable was a pipeline rather than a fixed interface. That is a genuinely hard thing to sell to a stakeholder counting incidents, and it cost me some credibility until the incident numbers moved.

Decision 03

Retry and dead-lettering redesigned as one decision

Considered: longer retry windows · higher retry counts · bounded retry with backoff plus a dead-letter route that alerts
Chose: A bounded retry with exponential backoff, and a dead-letter route with a named owner and a runbook.
Because: Unbounded retry is a way of hiding a failure rather than handling it — it converts a message that will never succeed into indefinite load and an alert nobody receives. Once retry is bounded, the only real question is how quickly a human finds out, which makes retry and dead-lettering one decision rather than two.
Gave up: A dead-letter queue without an owner is just a second silent failure, so this created an operational obligation that had not existed before. It only worked because the runbook and the rota were part of the same change.

Decision 04

Cost reduction as a by-product of the inventory, not a project

Considered: a dedicated cost-optimisation workstream · right-sizing as the inventory revealed what was unused
Chose: Decommissioning and right-sizing opportunistically, as instrumentation revealed what was actually running.
Because: The inventory was the expensive part, and it was already being paid for by the observability work. Once you can see which resources serve live traffic, the redundant ones identify themselves. A separate cost programme would have had to build the same picture again.
Gave up: The savings arrived gradually and unpredictably rather than against a forecast, which makes them harder to plan around.

Trade-offs accepted

Instrumenting first meant accepting a period where the reported failure rate rose sharply. That is politically expensive and it is the correct order anyway.
Automating releases before improving designs delayed every user-visible improvement by roughly a quarter.
Opportunistic cost work produced a real ~15% reduction but no repeatable process — a later engagement would need one.

Stack

Logic Apps
Azure Functions
Service Bus
Key Vault
Azure DevOps
RBAC
Log Analytics
KQL

All work