Evaluating RAG when you have no labelled dataset
Every team says they will build an evaluation set "once the product is stable". The product is never stable, and by then every change is a coin flip.
Start with thirty questions from real users
Pull them from support tickets, from the first week of internal usage, from the questions people asked in Slack before you built anything. Thirty is enough to catch a regression; three hundred is a project.
Score four things
- Groundedness — is every claim supported by a retrieved passage?
- Citation coverage — what share of claims carry a link?
- Retrieval hit rate — did the correct source appear in the top-k at all?
- Answer relevance — did it answer the question that was asked?
The first three can be judged mechanically or by a model. The fourth usually needs a person, which is another reason to keep the set small.
Separate retrieval failures from generation failures
If the correct passage never made it into the context, tuning the prompt is wasted effort. Retrieval hit rate is what tells you which half of the system to work on, and it is the cheapest metric to collect.
Run it on every change
An evaluation you run twice a year is a report. An evaluation that runs on every configuration change is a safety net. The value comes almost entirely from the second one.