We have been brought into six retrieval augmented generation projects that had stalled. In five of them the team was on their second or third retrieval strategy. They had moved from naive chunking to semantic chunking, added a reranker, swapped the embedding model, tried hybrid search.
In all five, when we asked how they knew the last change made things better, the answer was some version of: we tried a few questions and it seemed better.
That is the actual problem. Not chunking. Not the embedding model. The absence of any way to tell whether a change helped, which means every change is a coin flip and the team is doing a random walk through a large search space at considerable expense.
Why this happens to competent teams
Normal software has an obvious failure signal. The test fails, the endpoint returns 500, the page does not render. You know immediately.
An LLM system returns a fluent, confident, well formatted paragraph whether or not it is correct. The failure mode is indistinguishable from success at a glance. A demo of twenty questions will look fine even when accuracy on the real distribution is around sixty percent, because you unconsciously ask questions you know the system can answer.
So teams optimise the thing they can see, which is the pipeline, and ignore the thing they cannot, which is whether the output is right.
Build the golden dataset before the pipeline
The first artefact on any of our RAG projects is not code. It is a spreadsheet of questions with correct answers, built with the people who will actually use the system.
For a clinical policy assistant we built last year, that was 180 questions from three clinicians over two afternoons. Each row: the question in the words a real user would use, the correct answer, the source document and section that supports it, and a difficulty rating.
Two afternoons of clinician time. It is the highest leverage two afternoons on the entire project, because it converts every subsequent decision from an argument into a measurement.
Some things we have learned about building these:
Get the questions from users, not from engineers. Engineers write questions that match the document structure. Users ask "can I give this to someone on warfarin" when the document is titled "Anticoagulation Interaction Guidance". That gap is the whole retrieval problem and you will not discover it by writing your own test set.
Include questions the system should refuse. Perhaps fifteen percent of the set should be out of scope, unanswerable from the corpus, or built on a false premise. A system that confidently answers those is worse than useless in a regulated setting, and you cannot measure that behaviour if every question has an answer.
Record the supporting passage, not just the answer. This lets you measure retrieval and generation separately, which is the single most useful diagnostic you can have.
Version it. Ours live in the repository as YAML next to the code, reviewed like code. When someone adds a question because a user complained, that is a regression test forever.
Measure retrieval and generation separately
Once you have supporting passages, you can answer the question that unblocks most stalled projects: is the right information reaching the model at all?
Two numbers, computed independently:
Retrieval recall at k. In what share of questions does the retrieved context contain the passage that supports the correct answer? This is a pure information retrieval metric with no model involved. It is fast, cheap and deterministic.
Answer correctness given perfect retrieval. Feed the model the correct passage directly, bypassing retrieval, and grade the answer. This isolates the generation step.
The combination tells you exactly where to spend your time:
| Retrieval recall | Answer quality with perfect context | What is actually wrong |
|---|---|---|
| Low | High | Retrieval. Chunking, embeddings, hybrid search, reranking. |
| High | Low | Generation. Prompt, output format, model choice. |
| Low | Low | Your corpus probably does not contain the answers. |
| High | High | It works. Ship it and measure in production. |
That third row is more common than anyone expects. On one project the client was convinced they had a retrieval problem for four months. Recall at 10 was ninety one percent. The answers were poor because the source documents genuinely contradicted each other, having been written by different departments across six years. No amount of embedding tuning fixes a corpus that disagrees with itself. The fix was a content project, and it was the client's to do.
Automate the grading, but check the grader
For 180 questions run against every change, human grading does not scale. Using a language model as the grader works well, with one condition that teams skip: you have to validate the grader against human judgement.
Our process:
- Write a grading prompt that outputs a structured verdict with a short justification, not a bare score.
- Have a human grade 40 to 60 of the same examples independently.
- Measure agreement between the model grader and the human.
- If agreement is below about eighty five percent, fix the grading prompt, not the system under test.
We have had to iterate on grading prompts three or four times to get there. Common causes of disagreement: the grader rewarding fluency over accuracy, the grader accepting an answer that is technically true but omits a critical caveat, and the grader being harsh on formatting differences that no user would care about.
An unvalidated grader is worse than no grader, because it produces a number that feels like evidence.
Put it in continuous integration
Once the harness exists, it belongs in CI on every pull request that touches prompts, retrieval configuration or model selection. Ours reports a table:
Evaluation: policy-assistant (180 questions, claude-sonnet-4)
retrieval recall@10 0.94 (baseline 0.94, =)
answer correctness 0.89 (baseline 0.86, +0.03)
refusal precision 0.97 (baseline 0.97, =)
citation accuracy 0.91 (baseline 0.88, +0.03)
mean latency 2.10s (baseline 1.80s, +0.30s)
mean cost per query $0.0041 (baseline $0.0032, +28%)
4 regressions, 9 improvements PASS
Two things make this useful in practice.
Cost and latency sit next to quality. A change that improves correctness by three points and increases cost by twenty eight percent is a business decision, not an engineering one, and the pull request is where that conversation should happen.
Regressions are listed individually. Aggregate scores hide the case that used to work and now does not, and that specific case is often the one your most vocal customer asks about.
Then measure production, because your test set is wrong
The golden dataset is a fixed sample of a moving distribution. Real users ask things nobody anticipated, in worse English, about documents added last week.
Three things we instrument on every deployed system:
Log everything with a trace id. Query, retrieved chunks with scores, the full prompt, the response, model version, latency, token counts, cost. Storage is trivially cheap next to the cost of debugging blind.
A one click feedback control. Not a five star rating, which people ignore. A thumbs down with an optional free text box. Every thumbs down goes into a review queue.
A weekly triage. One engineer spends an hour reading negative feedback and sampling twenty random successful sessions. Anything interesting becomes a new row in the golden dataset. The test set grows to match reality instead of drifting away from it.
That last loop is what separates a system that gets better over eighteen months from one that plateaus in month two.
The order that actually works
If you are starting a retrieval project this week:
- Build the golden dataset with real users. Two afternoons.
- Build the harness that scores retrieval and generation separately. Two days.
- Validate your automated grader against human judgement. One day.
- Now build the simplest possible pipeline. Fixed size chunks, one embedding model, top-k, no reranker.
- Measure. Improve the number that is actually low.
- Wire the harness into CI and add cost and latency to the report.
- Ship, log everything, and grow the dataset from production every week.
Steps one through three take about a week and feel like a delay when leadership is waiting for a demo. That week is the difference between converging in a month and rebuilding retrieval three times over a year.
The most expensive thing in AI engineering is not the model. It is running experiments you cannot read the result of.