How do you know the AI is good enough?

In plain English

A test checks code. Given this input, the code must return this output. It passes or it fails.

An eval checks an AI model on a task where the answer can differ each time. We run the model on a set of examples, score the results, and see whether they fall inside the range we expect. An eval tells you how well the model does and how often, not just yes or no.

Google's paper The New SDLC With Vibe Coding (opens in a new tab) draws the same line. Tests verify the parts of a system that give the same answer every time. Evals verify the parts that do not. It also says that tests and evals are how you tell an agent what "correct" means, and that you write them before the code.

Where each fits in the software life cycle

Software teams already separate two questions. Are we building the product right? That is verification (opens in a new tab): does the build match its specification? Are we building the right product? That is validation: does it meet the real need? The pairing is credited to the software engineer Barry Boehm.

Automated tests are verification: they check the code against its specification. Evals are verification too, against a looser specification: a set of tasks and a rubric for a good answer. Neither tells you that you built the right agent. An eval gets close only when its tasks come from real use and real users judge the answers. Whether enough people need the agent at all is a product question. No test or eval answers it. That mapping is our reading. No source we read states it.

• While we build: tests and evals run in the same pipeline. Anthropic calls automated evals "the first line of defense".
• After launch: monitoring of real use shows where behavior drifts from what the evals assumed.
• Over time: people read transcripts, review samples and read user feedback. Their judgment calibrates the automated checks.

Anthropic compares this to the Swiss cheese model from safety engineering. No single layer catches every issue. Another layer may catch a failure that slips through. Ours are tests, evals, independent AI review and, where the risk calls for it, an engineer review. See who checks the AI's work.

How an eval is scored

These are the points from Anthropic's article that we think matter most.

• Three kinds of grader. Code checks a condition: a test passes, a lint check is clean. It is fast and repeatable, but it can reject a valid answer that looks different. An AI judge scores the work against a rubric. It is flexible, but it is not deterministic and needs calibrating against people. A person reading the work is the gold standard, and the slowest and most expensive. Anthropic's rule: code where possible, an AI judge where necessary.
• Check what the agent produced, not the steps it took. A flight agent may say it booked a seat. The outcome is whether the booking exists in the database. Anthropic calls checking one fixed sequence of steps too rigid.
• Run each task more than once. Say an agent solves a task 75% of the time on one try, and the tries are independent. Three tries in a row all succeed about 42% of the time (0.75 × 0.75 × 0.75). Anthropic names two measures: pass@k, at least one success in k tries, and pass^k, all k tries succeed. Use the first when one success is enough. Use the second when customers need consistency.
• Two kinds of eval. Capability evals start with a low pass rate, a hill to climb. Regression evals sit near 100% and tell you the agent still does what it did before. A task moves from the first kind to the second once it is solved.
• Start small, from real failures. Anthropic suggests 20 to 50 simple tasks drawn from real failures. A task is fair when two experts would reach the same verdict. A score of 0% across many tries more often means a broken task than a broken agent.

That last point is why the example below checks a check. In Anthropic's audit of a public coding benchmark, one task's tests assumed a file path that the task never stated.

An illustrative example: evals for an AI that drafts quotes

This example is made up to show the pattern. It is not a client case.

Say an AI agent turns a customer's enquiry into a draft quote: "We need 50 office chairs delivered to Munich by 1 December." The evals check each draft in three layers.

• Rule checks that need no AI. The quantity, the currency and the delivery date appear in the quote. VAT is stated, or the quote asks whether to include it. Every product and price matches the approved price list. Code checks these the same way every time.
• One judged check, against a written rubric. An AI judge reads the draft for one rule: it answers what the customer asked, in words fit to send.
• A person approves each quote. Nothing reaches the customer before that. We keep the AI draft, the check results and the person's edits together, so you can see later what the AI got wrong and what the person changed.

The order follows the rule above: code where possible, an AI judge where necessary. The person comes last because a quote commits your company to a price.

Tools you can start with

Two open-source tools can run evals like these. Promptfoo (opens in a new tab) tests prompts and agents from a simple config file. Its checks include contains and javascript for rule checks and llm-rubric for a judged check. Inspect AI (opens in a new tab), from the UK AI Security Institute, is a Python framework for evals of language models. Both are free under the MIT licence. We name them as starting points, not as an endorsement: we have not compared them with other tools.