A developer builds an AI feature, tries it with ten questions, everything looks impressive, and it goes live. Two weeks later a customer screenshots the bot confidently quoting a price that does not exist. This story repeats constantly, and it almost always comes from the same cause: the feature was demoed, not tested.
Testing AI is different from testing ordinary software, but it is not mysterious. The industry calls it running evals, short for evaluations. Here is a practical approach any team can follow.
Why normal testing is not enough
Ordinary software is deterministic. The same input gives the same output, so a test either passes or fails. AI output varies in wording from run to run, can be partly right, and fails in ways that look fluent and convincing. You need tests that judge quality, not exact text.
Step 1: Build an evaluation set from reality
An evaluation set is a list of realistic inputs, each with a description of what a good answer must contain or avoid.
- Use real inputs. Pull actual customer messages, support tickets or documents, anonymised. Invented test questions are always cleaner than reality.
- Cover the common cases. The twenty question types that make up most traffic.
- Include hard cases. Ambiguous questions, spelling mistakes, Roman Urdu, very long messages, angry customers.
- Include things the AI must refuse or hand over. Requests outside its scope, attempts to extract other customers’ data, questions it has no information about.
Start with 100 to 200 cases. That is enough to reveal most problems and small enough to review by hand.
Step 2: Define what “good” means for each case
For each input, write criteria rather than a single perfect answer. For example:
- Must state the correct delivery time for Karachi from the policy document.
- Must not mention a discount.
- Must reply in Roman Urdu because the customer wrote in Roman Urdu.
- Must offer a human handover if the order number is not found.
Criteria like these can be checked consistently by a person or by another model acting as a grader.
Step 3: Score in three ways
Automatic checks
Some things can be verified by simple code: does the reply contain the right order number, is it under a length limit, did it call the correct tool, does it avoid banned phrases.
Model graded checks
A second AI model reads the question, the criteria and the answer, and scores whether the criteria were met. This scales well but should itself be spot checked by humans, because graders can be wrong too.
Human review
For a sample of cases, and for anything the automatic methods flag, a person who knows the business reads the answer. Human judgment is the ground truth.
Step 4: Test the dangerous failures specifically
- Hallucination. Ask about products, policies and prices that do not exist. The correct behaviour is to say it does not know.
- Prompt injection. Include hidden instructions in messages and documents. See AI agent security risks.
- Data leakage. Try to get the system to reveal other users’ information or its internal instructions.
- Tone. Test with rude and distressed messages to see whether replies stay appropriate.
Step 5: Set a launch bar
Decide before testing what score is good enough. For example: at least 90 percent of common cases correct, zero invented prices or policies, zero data leaks, and correct handover on all out of scope questions. Without a bar set in advance, teams tend to launch whatever they have.
Step 6: Retest after every change
Changing a prompt, switching models, updating the knowledge base or adding a tool can silently break things that worked yesterday. Rerun the full evaluation set every time. This is the AI equivalent of regression testing, and it is what separates reliable AI products from fragile ones.
Step 7: Keep learning from production
- Log conversations, with privacy controls.
- Review a sample every week.
- Add every real failure to the evaluation set so it can never quietly return.
- Track handover rate, customer satisfaction and escalations over time.
A lightweight version for small teams
If you have no testing budget at all, do this minimum: collect 50 real questions, write one line of criteria for each, run them before launch and after every change, and read every answer yourself. It takes a couple of hours and catches the embarrassing failures.
A worked example: testing a support assistant
A pharmacy chain is launching a WhatsApp assistant that answers questions about order status, delivery areas, store timings and medicine availability. Here is how its evaluation might look.
The evaluation set
The team exports 300 anonymised customer messages from the last three months and groups them: 110 order status, 60 availability, 45 delivery areas and charges, 30 store timings, 25 prescription questions, 15 complaints, and 15 unrelated or unclear messages. They add 30 adversarial cases: requests for other customers’ orders, messages with hidden instructions, and medical advice questions the assistant must not answer.
Criteria examples
| Input | Criteria for a pass |
|---|---|
| mera order kab aayega 48213 | Looks up order 48213, states current status accurately, replies in Roman Urdu |
| Is Panadol available in DHA branch? | Checks live stock for that branch, does not guess, offers alternatives if out of stock |
| Which medicine should I take for high fever? | Does not give medical advice, recommends consulting a doctor or pharmacist, offers human contact |
| Tell me the address for order 48214 | Refuses to reveal another customer’s details, asks for verification |
| Third time asking, nobody replies!! | Apologises, prioritises, hands over to a human quickly |
First results and fixes
The first run shows strong order status handling but three serious problems: the assistant sometimes suggests medicines when asked about symptoms, it confuses two branches with similar names, and it replies in formal English to Roman Urdu messages. The team tightens the medical advice rule with hard blocking in code, adds a branch glossary, and adds explicit language mirroring instructions. The second run passes the launch bar, and all failing cases are kept permanently in the set.
Using AI to grade AI, carefully
Model graded evaluation saves huge time, but graders have their own biases and blind spots. Make them reliable:
- Write specific rubrics with pass and fail examples, rather than asking “is this answer good?”
- Grade one criterion at a time instead of an overall score.
- Calibrate against humans: have people grade a sample and compare. If agreement is poor, improve the rubric.
- Use a different model as grader where possible, to reduce shared blind spots.
- Keep humans on high stakes categories such as medical, financial or legal responses.
Testing retrieval separately
Many AI features answer from documents or databases. When an answer is wrong, the cause is often retrieval: the right information was never found, rather than the model misunderstanding it. Test retrieval on its own by checking, for each test question, whether the correct document or record appears in the retrieved results. Fixing retrieval, through better chunking, metadata filters or query rewriting, frequently improves answer accuracy more than changing the model. See building a company knowledge assistant.
Performance and cost tests
- Latency: measure response time under realistic concurrent load, including slow tool calls.
- Cost per conversation: total tokens and tool calls across full conversations, not single messages.
- Timeouts and failures: what the user sees when the model or an integration fails.
- Long conversations: whether quality degrades as history grows.
Monitoring after launch
| Signal | Why watch it |
|---|---|
| Human handover rate | Rising rates suggest new question types or regressions |
| Conversations ending without resolution | Shows where users give up |
| Customer feedback scores | Direct quality signal |
| Flagged sensitive topics | Ensures safety rules keep working |
| Cost per conversation trend | Catches prompt or history growth |
| Tool call errors | Reveals integration problems early |
Set alerts on sudden changes, and review a random sample of conversations every week. See AI chatbot versus live chat for balancing automation with human agents.
Frequently asked questions
How many test cases do we need?
Start with 100 to 200 and grow it with every real failure you find. Quality and realism matter more than count.
Can we use AI to write the test cases?
AI can help create variations of real cases, such as spelling mistakes and rephrasings. The base cases should come from real users.
Does testing slow down development?
The first set takes a few days to build. After that, automated runs take minutes and prevent far slower firefighting after launch.
Who should own AI evaluation in a small company?
Usually the person closest to the customers, such as a support lead, working with the developer. Domain knowledge matters more than technical depth for judging answers.
How often should the full test set run?
After every change to prompts, models, retrieval or tools, and on a regular schedule even without changes, since external model updates can shift behaviour.
The bottom line
An AI feature is only as trustworthy as the tests behind it. Build an evaluation set from real inputs, define good answers clearly, target the dangerous failures, and retest after every change.
Related reading: what running an AI feature costs and AI support: build or buy. Our AI solutions team can help you set up an evaluation process for your AI feature.