The model upgrade we didn't do

We had a retrieval-augmented research assistant in front of users, answering questions about market research and product claims over roughly 1.5 million indexed document chunks. Every answer carried citations and an explicit sufficiency verdict, because in a regulated domain “I’m not sure” is a valid and necessary answer.

A newer model became available in our Azure catalogue. The obvious move was to upgrade. We ran the comparison first, and the comparison said no.

What we measured

Answer quality in RAG is easy to assess badly. “Does this look right?” scales to about ten questions and then stops working. So the system was measured against a fixed 87-case evaluation set, with each answer judged for groundedness against its own cited evidence — not against a gold answer, but against whether the model’s claims were actually supported by the sources it chose to cite. That distinction matters: a fluent answer citing the wrong document is the failure mode you care about, and gold-answer matching won’t catch it.

The set was deliberately mixed:

• 71 fact-specific cases across brands, products, claims, and clinical studies.
• 16 behavioural cases, which are the ones people forget to write:
  • 8 out-of-scope questions the system must decline
  • 2 in-scope boundary questions it must answer — an over-refusal guard, because a system that declines everything scores wonderfully on safety and is useless
  • 3 requiring it to reach the external web source and rank that evidence correctly
  • 3 prompt-injection and jailbreak attempts it must refuse

The baseline

The deployed model scored 4.32/5 mean groundedness, median 5, with 97% of answers grounded and a 16% hallucination rate — meaning roughly one answer in six asserted at least one fact its own citations didn’t support.

Guardrails held: 8/8 out-of-scope declined, 3/3 injection attempts refused, 2/2 boundary questions correctly answered.

Strongest areas were open web research and single-product claim lookups. Weakest were cross-source synthesis and questions requiring numbers out of dashboard-style documents.

The candidate

The newer model scored 4.17 mean, 88% grounded, 19.7% hallucination.

Worse on every axis that mattered, and it over-declined — refusing questions the baseline answered correctly. For a research tool, over-refusal is not a safe failure. It teaches users the system is unreliable, and they stop asking.

Not adopted.

We also ran a frontier model from outside the Azure catalogue against the twelve baseline failure cases as a quality ceiling. It fixed all twelve. It was also undeployable for us — the platform’s data-residency boundary ruled it out entirely. Useful as a measurement, not as an option. Knowing the ceiling told us how much headroom existed; knowing the constraint told us we couldn’t reach it.

What the evaluation actually taught us

The hallucinations weren’t a model problem. Reading the failure cases individually, most were retrieval misses: the right source document was never fetched, so the model filled the gap. A clinical-study question would return market-research documents, and the model would answer from what it had. No amount of model capability fixes a system that hands it the wrong evidence.

That reframes the whole roadmap. The lever was retrieval and ingestion — chunking, reranker thresholds, how source routing decides which corpus to search — not model selection. We had been about to spend a migration’s worth of effort on the wrong layer.

Single runs are too noisy for small deltas. We tested one retrieval tweak that looked like an improvement and turned out to sit inside single-run variance. It wasn’t merged. Any future comparison of a small change needs 2–3 averaged runs, or you’re reading noise and calling it signal.

The part worth generalising

Most AI systems that disappoint in production are retrieval and data problems wearing a model costume. The instinct when quality is poor is to reach for a bigger model, because that’s the visible dial. It’s usually the wrong one.

The evaluation harness cost a fraction of what the migration would have, and it produced a defensible negative result — which is a real deliverable, not a failed experiment. It also left behind something durable: every future change to that system now has a baseline to be measured against.

Built on Azure AI Foundry with Azure AI Search agentic retrieval. Figures are from a fixed evaluation set run against the deployed system.