How to Evaluate AI Chatbot Accuracy Before and After Launch

If you are working out how to evaluate AI chatbot accuracy, stop asking "is it accurate?" and start measuring five separate things on a labeled test set of real questions: whether each answer is correct, whether every claim is supported by your own documents, whether anything important is missing, whether it refused questions it should not answer, and whether it escalated to a person when it should have. Grade those checks separately, by question category, with enough questions that the result means something, and set a written pass bar for each before launch. Then keep running the same tests after go-live, because the documents behind the answers keep changing.

That is the short answer. The rest of this article is the working method: how to build the test set, how to grade answers with code, humans and a judge model, how many questions you need before a score is worth trusting (most teams use far too few), what pass thresholds to set by risk level, and how to monitor accuracy once real customers are asking real questions.

It is written for the owner of the decision, a founder, COO, head of support or compliance lead, not only for the engineers. You do not need to run the tests yourself. You do need to know what a credible evaluation looks like, so you can tell whether the one in front of you is.

Why "it seems accurate" is not good enough

Most chatbot launches are approved after a demo. Someone senior asks a dozen questions, the answers read well, and the project ships. The problem is that fluent answers and correct answers look identical in a demo. NIST's Generative AI Profile names this risk confabulation: "the production of confidently stated but erroneous or false content (known colloquially as 'hallucinations' or 'fabrications') by which users may be misled or deceived." The confidence is the danger. A wrong answer that sounds hesitant gets checked; a wrong answer that sounds certain gets acted on.

The same NIST profile, published in July 2024 as NIST AI 600-1, lists as its first measurement action (MS-2.5-001) to avoid extrapolating system performance "from narrow, non-systematic, and anecdotal assessments." A dozen questions from an executive is exactly that kind of assessment.

Wrong answers now carry real liability

The legal picture has moved from theoretical to concrete.

• Moffatt v. Air Canada. On February 14, 2024, British Columbia's Civil Resolution Tribunal held Air Canada liable for negligent misrepresentation after its website chatbot told a grieving customer he could claim a bereavement fare retroactively, which the airline's actual policy did not allow. Air Canada argued, in effect, that the chatbot was responsible for its own words. The tribunal called this "a remarkable submission" and wrote: "It should be obvious to Air Canada that it is responsible for all the information on its website. It makes no difference whether the information comes from a static page or a chatbot." It ordered the airline to pay C$812.02. The sum was small; the principle was not.
• US consumer finance. The Consumer Financial Protection Bureau's June 2023 report on chatbots in consumer finance said chatbots must comply with all applicable federal consumer financial laws, and flagged three recurring problems: inaccurate answers, "doom loops" with no route to a human, and failure to recognize when a customer is raising a dispute.
• Unsubstantiated AI claims. In September 2024 the FTC announced its crackdown on deceptive AI claims, including a $193,000 settlement with DoNotPay, which marketed a "robot lawyer" that, according to the FTC, it had never tested against the performance of a human lawyer. Then-Chair Lina Khan said there is "no AI exemption from the laws on the books."

Notice the pattern. In every case the failure was not "the model was imperfect." It was that nobody measured whether the answers were right before customers relied on them. An evaluation record is how you show you took reasonable care. If your chatbot touches regulated decisions, involve your AI compliance lead and counsel early; nothing here is legal advice.

Key takeaway: A chatbot's answers are your company's statements. The question is not whether it will ever be wrong, but whether you measured how often, on what, and did something about it.

What does "accuracy" actually mean for a chatbot?

"Accuracy" is five different failure modes wearing one word. They have different causes, different fixes and different costs, so they must be measured separately. A single blended "92 percent accurate" number hides which one is hurting you.

The Stanford study of legal research tools makes the second row concrete. Researchers ran more than 200 legal queries through commercial tools built specifically to avoid hallucinations and found that Lexis+ AI and Ask Practical Law AI gave incorrect information more than 17 percent of the time, and Westlaw's AI-Assisted Research more than 34 percent. They counted two kinds of hallucination: answers that were simply wrong, and answers that were correct but cited a source that did not support them, which they called misgrounded. Thomson Reuters disputed parts of the methodology, and the products have changed since that May 2024 study, but the lesson holds: retrieval alone does not make answers trustworthy, and citations need checking too.

For a finance team the "unsupported claim" row has a sharper edge: a language model asked to produce a number it was not given will often produce a plausible one. We return to that below.

Step 1: Build a test set from real questions

Everything depends on the test set, sometimes called a golden set: a fixed list of questions, each with the expected answer and the document that supports it. Without one, every change to the system is a matter of opinion. With one, you can say "the new chunking strategy raised groundedness from 91 to 96 percent and broke two refund questions," which is a decision someone can make.

OpenAI's evaluation best practices recommend drawing eval data from production logs, expert-written references, historical logs and synthetic cases, and covering typical, edge and adversarial inputs. Anthropic's guidance on building evaluations says to mirror your real-world task distribution and not to forget the edge cases: irrelevant or nonexistent input, overly long input, poor or hostile input, and ambiguous cases where humans would also disagree.

Where the questions come from

1. Your ticket history. Export the last 6 to 12 months of support tickets or chat logs, cluster them by topic, and sample proportionally. If 30 percent of tickets are about billing, roughly 30 percent of the test set should be too. This is the single most valuable source because it reflects what customers actually ask, in the words they actually use.
2. High-risk topics, over-sampled. Anything where a wrong answer costs money, creates a legal commitment or harms someone: prices, refunds, cancellations, eligibility, contract terms, safety, medical or financial facts. Add extra questions here beyond their natural share.
3. Unanswerable questions. Questions your documents do not cover, so the correct response is a clear "I don't know, here is how to reach someone." Ten to fifteen percent of the set is a sensible share.
4. Escalation triggers. Disputes, complaints, threats, mentions of self-harm, requests for a human, and anything regulated that must go to a person.
5. Adversarial inputs. Attempts to get the bot to promise discounts, reveal its instructions, ignore its rules, or talk about competitors.
6. Paraphrases. The same question asked three different ways, including with typos and slang. Consistent answers across paraphrases are a cheap signal of reliability.

What each test case should contain

The reference answers must be written or approved by the people who own the policies, not by the engineers building the bot. This is where most of the effort goes, and it is also where most of the value is: building the test set routinely surfaces contradictions between documents that were invisible until someone had to write down the one correct answer.

Step 2: Measure retrieval separately from the answer

Most business chatbots use retrieval-augmented generation (RAG): they search your documents for relevant passages and then ask the model to answer from those passages. When an answer is wrong, the cause is in one of two places, and the fix is completely different for each.

Microsoft's documentation for its RAG evaluators gives a clean public vocabulary for the split, which you can use whatever platform you build on:

• Process evaluation asks whether retrieval found the right passages. With human relevance labels you can compute search metrics such as NDCG (whether the best documents are ranked at the top) and fidelity (how many of the known good documents came back). Without labels, a judge model can score how relevant the retrieved chunks look.
• System evaluation asks whether the final answer was good. Microsoft describes groundedness as the precision side (the response contains nothing outside the grounding context) and response completeness as the recall side (it does not miss critical information compared with the expected answer).

The open-source Ragas library defines a similar measure, faithfulness, in a way anyone can understand: break the answer into individual claims, check whether each claim can be inferred from the retrieved context, and divide supported claims by total claims. An answer with four claims, three of them supported, scores 0.75. It is a useful mental model even if you never use the library, because it turns "the bot made something up" into a number you can track.

Here is why the split matters in practice. If retrieval is finding the right passage and the answer is still wrong, you work on the prompt, the model or the answer format. If retrieval is not finding the right passage, no amount of prompt work will help; you work on how documents are split, labeled, searched and ranked. Teams that only grade final answers spend weeks tuning prompts to fix what is really a search problem. Our breakdown of RAG chatbot development cost shows how cheap the retrieval fixes usually are compared with the engineering time wasted on the wrong layer.

Step 3: Grade answers with code, humans and a judge model

There are three ways to grade an answer, and a good evaluation uses all three for different checks.

Code checks first

Anything that can be checked with code should be. If the correct refund window is 30 days, a test that fails whenever the answer contains "60 days" or lacks "30" is more reliable than any model. Anthropic's guidance puts it bluntly: structure questions to allow automated grading wherever possible, and prefer more questions with slightly lower-signal automated grading over fewer hand-graded ones. The same applies to behavior: if a conversation should trigger a hand-off, check the log for the hand-off event, not the wording.

Using a model to grade a model

For the checks that need judgment, such as "is every claim in this answer supported by these passages?", a second language model can grade at scale. The research basis is reasonable. The 2023 paper Judging LLM-as-a-Judge by Zheng and colleagues found that strong judge models agreed with human preferences more than 80 percent of the time, about the same rate at which humans agree with each other. The same paper documented three biases you must design around: position bias (favoring whichever answer is shown first), verbosity bias (favoring longer answers) and self-enhancement bias (favoring answers written by the same model).

Practical rules that follow from that, consistent with OpenAI's and Anthropic's published guidance:

1. Use pass/fail or pairwise comparisons, not 1-to-10 scores. OpenAI notes these are more reliable than open-ended scoring. "Is every claim supported: yes or no, and quote the unsupported claim" is far more consistent than "rate the groundedness."
2. Use a different model to grade than the one that answered. Anthropic's examples recommend this explicitly.
3. Calibrate before you trust it. Have two subject experts grade 100 to 200 answers independently, measure how often they agree with each other, then tune the judge's instructions until it agrees with them about as often. OpenAI's guidance is to scale up only once the judge is faster, cheaper and consistently agrees with human annotations.
4. Keep auditing. Re-grade a random 5 to 10 percent of the judge's verdicts by hand every cycle. Judges drift when the underlying model is updated.

Where humans stay in the loop

Humans write the reference answers, calibrate the judge, and personally review every high-risk category before launch. Budget for it. Two experts spending a few days building and grading a 300-question set is a small cost against the price of the first wrong refund promise.

How many test questions do you need?

This is the part almost every guide skips, and it is the reason so many "95 percent accurate" claims are meaningless. Accuracy measured on a sample is an estimate, and small samples give wide ranges. The NIST/SEMATECH e-Handbook describes the standard way to put a confidence interval around a proportion; it recommends the Wilson method, partly because the simple textbook formula can produce impossible values when samples are small or failures are rare, which is exactly the chatbot case.

We applied the Wilson method at 95 percent confidence to a chatbot that scores 95 percent correct on test sets of different sizes:

Read the first row carefully. A chatbot that gets 19 of 20 demo questions right could plausibly be wrong one time in four. The last column matters even more for high-risk topics: if you test 20 refund questions and the bot gets all 20 right, the true error rate on refund questions could still be as high as 16 percent. "No failures in testing" means far less than it sounds like on small samples.

What this means in practice:

• A floor of 200 to 400 labeled questions for the overall launch decision. Below that, you cannot tell 92 percent from 97 percent.
• At least 20 to 30 questions in every high-risk category, and accept that even a perfect score there only rules out failure rates above roughly 10 to 15 percent. That is why high-risk topics also need structural protection (forced escalation, approved answer templates), not only testing.
• Compare versions on the same test set. When you change something, run the same questions before and after. Differences of a point or two on a few hundred questions are often noise; look for consistent changes in specific categories.
• Always report the sample size next to the score. A vendor or internal team that reports "96 percent accuracy" without saying "on how many questions, in which categories" has not told you anything yet.

What accuracy bar should you set before launch?

There is no regulatory or industry standard number for chatbot accuracy, and anyone who quotes one is guessing. The right bar depends on what a wrong answer costs in your business. What you can do is make the bar explicit, tie it to risk, and get it signed off by whoever owns that risk before the build is done, so "good enough" is not decided under launch-week pressure.

The scorecard below is the framework we suggest as a starting point. The thresholds are recommendations, not standards; adjust them with your compliance, legal and support leads.

Three principles sit behind the table.

First, unanswerable is a valid answer. Score the bot on whether it declines correctly, not only on whether it answers. Anthropic's guidance on reducing hallucinations lists explicitly allowing the model to say "I don't know" as its first technique. A bot that answers 80 percent of questions correctly and declines the rest gracefully is usually better for the business than one that answers 100 percent and is wrong on 8 of them.

Second, track the answer rate next to accuracy. A bot can raise its accuracy by refusing more. Report both, so nobody "improves" the score by making the bot useless.

Third, some topics should never be left to generation. For Tier 3 questions, the safest design is often to detect the topic and return pre-approved wording or hand off to a person. Measuring accuracy on those topics then becomes a test of the detection, which is much easier to get to near-perfect than free-form answers.

What public hallucination benchmarks can and cannot tell you

Leaders often ask which model hallucinates least, hoping a better model will solve the problem. Benchmarks help a little, and only a little.

Vectara's public Hallucination Leaderboard, last updated September 22, 2026, measures how often models introduce unsupported information when summarizing documents, across more than 7,700 articles. On that test, listed models ranged from 1.8 percent for the best performer to more than 20 percent for some well-known reasoning models, and popular general-purpose models from every major provider sat between roughly 3 and 12 percent. Two things are worth noticing. The spread between models is real, so model choice matters. And even the best models are not at zero on a task as simple as summarizing a document that is right in front of them.

But the benchmark is not your chatbot. It measures one task, on one dataset, with one scoring model. Your system's accuracy depends at least as much on retrieval quality, document quality, prompt design and the mix of questions your customers ask. That is the point of NIST's warning about extrapolating from narrow assessments. Use benchmarks to shortlist two or three models; use your own test set to choose between them. Often the cheaper model wins on your data, which also helps the bill, as our guide to LLM API cost optimization explains.

Why AI makes up numbers, and how to stop it

The second question in our research came from finance teams: AI tools inventing figures in reports and reconciliations. This is a specific and predictable failure, and it has a specific fix.

Language models generate text one piece at a time, choosing what is likely to come next. Ask one for a revenue figure that is not in its context, and a plausible-looking number is a likely next piece of text. The model is not lying; it has no built-in way to tell "a number I read" from "a number that fits." The CFPB report makes the same point about large language models in general: they are not well positioned to distinguish factually correct from incorrect data.

The fix is architectural, not a better prompt:

1. Never let the model produce a number it did not receive. Numbers come from a database query, a spreadsheet calculation or an API, executed by code. The model's job is to choose which query to run and to explain the result in words.
2. Have the model cite the source of every figure, and check with code that each cited figure actually appears in the source. Anthropic's guidance recommends exactly this pattern: have the model find a supporting quote for every claim and retract any claim it cannot support.
3. Test it. Add questions to your test set that ask for numbers the system does not have. The only passing answer is "I don't have that figure."
4. Log everything. For any number that reaches a decision, you need to show where it came from. Our guide to AI audit trail requirements covers what to record.

The same discipline applies to customer support. A bot that quotes a price should be pulling it from the pricing system at that moment, not recalling it from a document that was correct last quarter.

How to keep measuring accuracy after launch

Pre-launch testing tells you how the bot performs on questions you thought of. Production tells you how it performs on the questions customers actually ask, and the two drift apart within weeks. NIST's profile calls for evaluating performance in real-world conditions to reveal issues that do not surface in controlled testing (MS-4.2-002), and for reviewing and verifying sources and citations during ongoing monitoring, not only before deployment (MS-2.5-003).

A monitoring routine that works for most mid-sized teams:

• Automated regression on every change. Re-run the full test set whenever the prompt, model version, retrieval settings or knowledge base change. OpenAI's guidance calls this continuous evaluation. Block the release if any high-risk test fails.
• Scheduled re-runs even when nothing changed. Source documents get edited, links break, model providers update models behind the same name. A weekly automated run catches drift you did not cause.
• Weekly human review of a random sample. Have a subject expert read 50 to 100 randomly selected live conversations a week, plus every conversation a customer rated badly or a human agent took over. Grade them with the same five checks as the test set.
• Turn every failure into a test. Each wrong answer found in production becomes a new test case, so it cannot silently come back. Over a year, the test set becomes a precise map of how your customers actually trip the system.
• Watch the proxy signals. Escalation rate, repeat contacts within 48 hours, "that's not what I asked" replies, thumbs-down ratings and sudden changes in conversation length. None proves an answer was wrong, but a shift in any of them tells you where to look.
• Re-baseline when documents change. When a policy changes, update the affected reference answers on the same day, or your tests will start failing correct answers and passing wrong ones.

This is ongoing work, which is why it belongs in the budget from the start. The difference between a chatbot that keeps its quality and one that slowly degrades is rarely the model; it is whether someone owns this routine. Our article on moving an AI pilot to production covers the ownership question in more depth.

How to evaluate AI chatbot accuracy in your first 30 days

Put together, a credible evaluation for a mid-sized company's support chatbot looks like this:

If a vendor is building the system, ask to see this plan in their proposal, ask who writes the reference answers, and ask for the test set to be delivered to you as part of the project. A test set you own lets you hold any future vendor, model or version to the same standard. Our guide to build vs buy AI agents covers how to compare packaged products against custom builds; the evaluation method here works for both. If you are considering a voice bot rather than chat, the same checks apply, with extra attention to misheard inputs; see our guide to the AI receptionist for small business.

When not to launch

Hold the launch, or narrow its scope, if any of these are true: high-risk categories have fewer than 20 tests each; the bot fails any must-not-say test; nobody outside engineering has reviewed the reference answers; there is no tested route to a human; or nobody is named as the owner of post-launch monitoring. A narrower bot that handles three categories well is a better first release than a broad one that is wrong in ways nobody has measured.

Where this leaves you

Evaluating AI chatbot accuracy is not a single number and not a one-time event. It is a labeled test set built from real customer questions, five separate checks graded by code, a calibrated judge and human experts, enough questions to make the score meaningful, a written pass bar tied to risk, and a monitoring routine that keeps running after launch. None of it requires exotic technology. It requires deciding, before launch, what "correct" means and who is responsible for it.

If you are planning a chatbot or have one that customers are starting to rely on, the most useful next step is usually to build the test set before anything else, because it tells you what the system needs to be. Fleurant AI builds custom AI and generative AI systems with evaluation built in from the first prototype, which we deliver in one to two weeks. If you would like a second opinion on a chatbot's accuracy plan, or on a vendor's claims, talk to a specialist and you will hear back within one business day. You can also start from the Fleurant AI home page to see the other services.