To evaluate an LLM application, collect a set of real inputs with the answers or behavior you expect, score the application’s outputs against them with a mix of automatic checks, reference answers, LLM judges and human review, and run that evaluation on every change to prompts, models or data. The score tells you whether a change made things better or worse before your users find out.
Key takeaways
- Evaluate the whole application, not just the model: retrieval, prompts, tools and guardrails all affect the answer.
- Build the evaluation set from real cases, including the hard and unusual ones.
- No single scoring method is enough. Combine cheap automatic checks with LLM judges and regular human review.
- Treat the evaluation like tests in software: it runs before every release and blocks regressions.
Why ordinary tests aren’t enough
Traditional software tests compare an output with one exact expected value. LLM applications produce open-ended text, so two different answers can both be right, and a fluent, confident answer can be wrong. Quality also has several sides at once: is the answer correct, is it supported by the sources, is it complete, does it follow policy, is it in the right format? An evaluation has to measure the sides that matter for your use case.
Step 1: Define what good looks like
Write down, with the people who own the outcome, what a good answer or action is. For a support assistant that might be: correct, supported by the help center, under 120 words, polite, and handed to a person when the question is about a refund. These criteria become your scoring rules.
Step 2: Build an evaluation set
An evaluation set is a list of inputs with the expected answer, the expected source or the expected behavior.
- Start from real traffic: past tickets, emails, searches or chat logs, anonymized where needed.
- Cover the spread: common questions, rare ones, questions with no answer in your content, ambiguous ones and ones users phrase badly.
- Add adversarial cases: attempts to make the system ignore its instructions, reveal data or act outside its role. The OWASP Top 10 for LLM applications (2025) is a good checklist of what to try, starting with prompt injection.
- Have experts mark the answers. The set is only as good as its labels.
Start with a few hundred cases if you can; a smaller set that covers the important cases is better than none. Add every real failure you find after launch.
Step 3: Choose how to score
Use the cheapest method that measures each criterion reliably:
| Method | Good for | Watch out for |
|---|---|---|
| Rules in code | Format, length, required fields, banned phrases, valid JSON | Only checks what you can write as a rule |
| Reference comparison | Questions with one known right answer or source | Penalizes correct answers phrased differently |
| LLM as a judge | Correctness against sources, helpfulness, tone, policy | Judges have biases; check them against people |
| Human review | Final say on quality, especially in regulated work | Slow and costly; use it on samples |
| Production signals | Thumbs up or down, handoffs, edits users make | Arrive after launch and are noisy |
LLM judges deserve care. Zheng and colleagues (2023) found that strong LLM judges agreed with human preferences about as often as humans agreed with each other, at over 80 percent in their tests, but also measured position, verbosity and self-enhancement biases. Write clear grading instructions, test the judge against a set your experts have scored, and re-check it when you change the judge model.
For retrieval-based applications (RAG), score the retrieval separately from the answer: did the system find the right passages, and did the answer stay faithful to them? Frameworks such as Ragas (Es and colleagues, 2023) were built for this.
Step 4: Run it on every change
Wire the evaluation into your release process:
- Run the full set on every change to a prompt, model, retrieval setting or tool.
- Compare scores with the last release, per criterion and per category of question.
- Block the release if a key score drops below the agreed threshold.
- Keep the results, so you can see quality over time and explain changes.
Step 5: Keep watching in production
After launch, sample real conversations for human review, track handoffs and user feedback, and watch cost and latency. Turn every confirmed failure into a new evaluation case. The NIST AI Risk Management Framework (AI RMF 1.0, 2023) is a useful structure for deciding who reviews what and how often.
A worked example
Example, for illustration. A knowledge assistant for an HR team:
- Set: 300 real employee questions from the HR inbox, each with the policy section that answers it, plus 40 questions the handbook doesn’t answer and 20 attempts to get personal data about colleagues.
- Scores: retrieval hit rate (was the right section retrieved), faithfulness (an LLM judge checks the answer against the retrieved text), correct refusals on the unanswerable and unsafe cases, and a weekly human review of 30 random production answers.
- Release rule: no release if faithfulness or correct refusals drop compared with the previous version.
Common mistakes
- Testing on made-up questions only. Real users ask differently from the team that built the system.
- One overall score. An average can hide a collapse in one category, such as refusals.
- Trusting the judge blindly. An unchecked LLM judge can reward long, confident answers.
- Evaluating once. Models, prompts and content change. An evaluation that doesn’t run on every change stops protecting you.
How KaliIO approaches it
We write the evaluation set with your experts before we tune anything, and we keep it in the repository next to the code, so it runs in CI with every release. Our LLM evaluation services cover evaluation sets, judges, red-teaming and regression checks; MLOps and LLMOps covers running them in production; and every LLM application we build ships with its evaluation.
Frequently asked questions
How many test cases do I need?
Enough to cover the categories of input that matter, with several examples each. A few hundred is a practical start for most applications. Grow the set with every real failure.
Can I use the same model to generate and judge answers?
You can, but models tend to rate their own outputs more favorably, one of the biases Zheng and colleagues describe. Use a different judge model where possible, and always check the judge against human scores.
What is red-teaming?
Deliberately trying to make the system fail: ignore its instructions, reveal data, produce harmful content or take actions outside its role. Red-teaming cases belong in the evaluation set so they are re-tested on every release.
Do I still need normal software tests?
Yes. Interfaces, APIs, permissions and integrations still need automated tests. Evaluations cover the parts whose output is language or judgment.
Sources
- Zheng, L. et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.
- Es, S. et al. (2023). Ragas: Automated Evaluation of Retrieval Augmented Generation.
- OWASP Gen AI Security Project (2025). OWASP Top 10 for LLM Applications 2025.
- NIST (2023). AI Risk Management Framework (AI RMF 1.0).