Every project I take on hits the same moment. The demo works, everyone is pleased, and someone asks whether we can put it in front of real users next week. The answer is always the same: not until we can tell whether it's getting better or worse.
That's not caution for its own sake. Without a measurement, prompt changes become a kind of office folklore. Someone tweaks the system prompt because a support ticket mentioned a weird answer, the fix quietly breaks four other cases, and nobody finds out for a month.
Start with thirty real examples
Not synthetic ones. Real inputs, copied from the actual source: real emails, real documents, real questions people have asked. Synthetic test data is written by the same brain that wrote the prompt, so it tests the same assumptions.
I aim for around thirty to start. If a task has obvious categories, I spread them across the categories rather than picking thirty variations of the easy case. Half a day of work, usually less, because the client already has the material.
Each example gets:
- The input, unchanged apart from removing anything identifying.
- What a good answer looks like, in prose. Not a score, an actual answer.
- A short note on what would make a response unacceptable.
Grade the shape, not just the wording
Anyone who has tried to match model output to an exact string knows it fails on a trivial rewrite. So I check two layers separately.
The first is deterministic: did it call the right tool, in the right order, with valid arguments? Did it include the required fields? Did it refuse where it should have refused? These are assertions, and they either pass or they don't.
The second layer is judgement, and I use a model to grade it against the reference answer with a strict rubric: does the response contain every required fact, does it contradict the source, does it invent anything. Model grading is imperfect, which is exactly why it's the second layer and not the first.
The number that matters is the delta
Nobody should care whether a run scores 82% or 86% in the abstract. The useful signal is directional and comparative:
Last week's prompt scored 24 of 30. This week's scores 29. It also costs 40% less because it stops after three tool calls instead of seven.
That sentence is what lets you ship a change with confidence, and it's what stops a well-meaning edit from shipping blind.
Wire it to the commit
An evaluation set that lives in a notebook is decoration. It belongs in CI, running on the paths that touch prompts, retrieval configuration or tool definitions. It doesn't need to block every pull request, but it should make a regression visible within minutes rather than during a customer call.
I keep a short failure log too, one line per real-world failure, appended as it happens. Those lines become the next ten test cases, and they're worth more than anything I could invent.
The uncomfortable part
Writing the evaluation set forces a conversation about what "good" actually means. Teams often disagree about this and didn't know it. One person thinks a shorter answer is better, another wants the full audit trail. Better to find that out in an afternoon of writing test cases than three weeks into a build.
If you take one thing from this: the evaluation set is not a quality gate you add at the end. It's the specification. Write it early, and the rest of the project gets noticeably less vague.
