Evals are user research at machine speed

Most teams I talk to think of evals the way they think of tests. A gate at the end. Something an engineer sets up once and nobody looks at again until it goes red.

I think that framing wastes the most useful discovery tool we have picked up in years.

The reframe

An eval set is a list of things you believe your product should be able to do, written down, with the right answer attached. Read that sentence again. That is a product spec. It just happens to be executable.

When I sit down with a founder and we build one together, the argument that happens is never about tooling. It is about what "correct" means. Should the assistant answer a question when it only has partial evidence? Should it cite one source or four? If the customer asks something outside the docs, is silence a success or a failure?

Those are product decisions. Building the eval set forces them into the open on day one instead of month four.

Start with forty

You do not need a framework. You need a spreadsheet with a question, an ideal answer, and the document the answer should come from.

Forty rows. It takes an afternoon. Bias it hard toward:

• Questions where the honest answer is "we do not have that information"
• Questions two customers phrased completely differently
• The three questions your support team is sick of answering
• Anything involving a number, a date, or a price, because those fail loudly and get you fired
• One or two genuinely hostile inputs

The unanswerable ones matter most and almost everyone skips them. A system that answers 100% of questions is not good, it is a system with no abstention policy.

The medical study that changed how I think about abstention

There is a study where clinicians reviewed six chatbot configurations against 62 cancer information questions. Retrieval grounded in a controlled, curated source dropped medically harmful hallucinations to 0% for the strongest setup, against roughly 40% with no retrieval at all.

Huge result. But here is the bit I keep coming back to: those same RAG systems only answered 36% to 81% of the questions. The models without retrieval answered 100% of them, confidently, and were dangerous.

So the retrieval system looked worse on the metric most dashboards track and was obviously, hugely better. Answer rate is not quality. Sometimes a lower answer rate is the feature. If you are not measuring abstention alongside accuracy, your dashboard is lying to you in a direction that feels good.

Measure the retriever separately

The mistake I made for a long time was scoring only the final answer. The problem is that a good answer can hide bad retrieval, because the model already knew the fact from pretraining. That works right up until you ask about something proprietary, and then it collapses and you have no idea why.

Score them apart:

• Retrieval: was the right document in the top k at all
• Faithfulness: does every claim in the answer trace to something retrieved
• Abstention: when evidence was missing, did it back off

Three columns. You can eyeball the failures over coffee and you will learn more than from a month of vague "the AI feels off" feedback.

The compounding bit

Every bug report becomes a row. Every weird support ticket becomes a row. After six months you have a description of your product's actual job, written in the language of your actual users, that any new engineer can run in ninety seconds.

Prompts do not compound. Model choices definitely do not compound, they expire. The eval set compounds.