A demo answers one question: can this agent do something impressive on a case somebody picked for it. Production asks a different question. What evidence exists that it produces a verifiable outcome on your real cases, run after run — and stops cleanly when it doesn’t know?

Until you have that evidence, what you have is a prototype. Acceptance testing is how you close the gap without waiting for production incidents to find out.

AI agent evaluation is the set of tests and acceptance criteria that turns that business question into an organised check: representative cases, a defined expected outcome, forbidden actions, observable traces, and an explicit decision at the end.

Why a demo is not release evidence

Prototypes convince because they run under favourable conditions: the question is clear, the data is clean, the tool responds. Anthropic’s engineering guidance names the structural problem: the capabilities that make agents useful — autonomy, tool use, adapting mid-task — are exactly the ones that make them hard to evaluate, and the strategies that hold up in real deployments combine several kinds of testing rather than one score.[1]

Concretely, a demo fails on three counts:

  • it hides variance. Run the same case ten times and you will likely see different trajectories;
  • it skips your ambiguous, incomplete and out-of-scope cases, which are most of real traffic;
  • it never shows the failure mode that matters: what happens when a tool times out, a field is missing, or the request falls outside the agent’s lane.

The decision evaluation exists to inform

Before any rollout, you should be able to answer: what evidence exists, on which cases, against which release blockers — and who signs the go, rework or no-go?

Scope an AI agent with Last Word

Write the eval contract

The central document fits on one page. Not a technical spec — a contract between the business sponsor and whoever ships the agent. For each scope under test, it fixes:

  • the task: what the agent must accomplish, in business language;
  • the input: a representative case, with its documents or data;
  • the expected end state: what should exist after the agent runs — a triaged draft, a filled field, a sourced answer — and what must not have changed;
  • the forbidden actions: send, delete, publish, pay, change permissions;
  • the release blockers: the failure families that rule out any rollout, however well everything else passes.

The contract does not describe how the agent works. It describes how you will know whether it works. That separation is also what distinguishes it from a functional brief — the AI support agent brief we describe elsewhere covers product and operating requirements. Building the prototype and its scoping come first; acceptance testing judges the result against evidence.

One-page AI agent eval contract: normal, ambiguous and forbidden cases on the left, observed tool trace with stop points in the middle, release blockers and the bounded GO / limited pilot / no-go decision on the right.

Build a test case library that earns its keep

A useful library is not a pile of easy wins. It deliberately covers what actually arrives:

  • normal cases: the well-formed, in-scope request;
  • ambiguous cases: two defensible readings, unclear priority;
  • incomplete cases: missing information, empty field, truncated document;
  • out-of-scope cases: legitimate requests that are simply not this agent’s job;
  • tool failures: the source times out, the API errors, the data is unavailable;
  • untrusted content: external text that carries instructions — indirect prompt injection deserves its own test family.

Every case is fictional or de-identified, written from your operating reality, never copied from a live flow. NIST is building in this direction with evaluation probes embedded in agentic workflows, whose results accumulate into a machine-readable audit trail for checking the factual grounding of outputs — a support for verification, not a substitute for the human who accepts or refuses.[2]

Score the outcome and the trajectory

For an agent, the end state is not enough. Two runs can land the same final state by very different paths: one citing a reliable source, one improvising. The trajectory — the end-to-end record of model calls, tool calls and guardrails, as OpenAI defines the trace — is what makes the path observable.[4]

For each case, record at minimum:

  1. the business outcome: does the end state match the contract?;
  2. the tool choice: the right tool, or a workaround?;
  3. the arguments and sources: is the outcome grounded, or fabricated?;
  4. escalations: was the ambiguous case handed to a human, or decided alone?;
  5. stopping behaviour: does the agent halt cleanly at a failure or a boundary?;
  6. what actually changed: which systems were touched beyond the announced deliverable?

Anthropic’s vocabulary is worth keeping straight here: a task with defined success criteria, a grader that scores an aspect of performance, and the trace of what actually happened.[1] What to store, alert on and dashboard once the agent runs in production belongs to observability, which is a separate downstream concern.

Match each criterion to the right grader

Not every check is scored the same way. The NIST AI RMF playbook asks teams to select measurement approaches against the risks that matter, document acceptable limits, and also declare what cannot be measured.[3]

Three grader families coexist, with different strengths:

  • deterministic rules: the field is present and well-formed, the destination is on the allowlist, no forbidden action appears in the trace. Automatable, unambiguous — the default choice wherever it fits;
  • model-as-judge: a second model scores a quality that is hard to code — tone, relevance, completeness. Useful, but probabilistic. Anthropic recommends calibrating it against human judgement for subjective outputs, and it should never be presented as independent, objective or sufficient on its own;[1]
  • human review: for final acceptance and edge cases. Slower, and the only layer that carries the responsibility of the decision.

The failing eval design routes everything through a model judge because it is cheaper. The sound one assigns a named grader to every criterion in the contract — and puts the release blockers on deterministic checks first. Microsoft’s guidance frames the same idea from the business side: systematic evaluation turns a vague feeling (“the agent isn’t working well”) into a localised observation tied to objectives and pre-release regression tests.[5] It is vendor documentation with flattering examples — keep the articulation, not the sample numbers.

Classify failures instead of averaging them away

A mean pass rate hides the information you need. “90 % success” can mean a few cosmetic misses, or one run in ten performing a forbidden action. Those are not the same risk profile.

Sort every failure into a family:

  • fabricated fact: the outcome contains unsupported or false information;
  • wrong source: the right answer existed, the agent cited something else;
  • wrong tool: the appropriate tool was available, the agent picked another;
  • forbidden action: the trace shows a send, a delete, an out-of-scope change;
  • missing escalation: an ambiguous case was decided instead of handed over;
  • loop: the agent repeats or stalls without stopping;
  • unusable output: technically successful, unusable by the business.

The classification changes the decision. A low failure rate concentrated on forbidden actions or missing escalations blocks a rollout harder than a higher rate of poorly worded drafts. You do not pilot in the same place, or with the same scope.

Compare baseline and candidate, versions explicit

An agent is never evaluated in a vacuum: it is compared. Against the previous version, the manual process, or a prompt variant. The comparison only means something if the eval contract stays strictly identical: same cases, same graders, same blockers.

Record explicit versions of everything that can shift behaviour: model, prompt, exposed tools, data sources. A regression test answers a different question from a capability test: it does not measure what the agent can newly do, it checks that an acquired behaviour did not disappear.[1] OpenAI formalises the same progression — inspect individual traces while debugging, then move to repeatable datasets and eval runs once expected behaviour is defined.[4]

One reading rule: never let an average hide the critical cases. If the candidate improves the global score but regresses a blocker case, the finding is the regression.

Take a bounded decision: GO, REWORK or NO-GO

The evaluation ends with a decision, not a report. Three outcomes are enough:

  • GO: the blockers pass, residual failures are classified and acceptable. The rollout starts limited, with an explicit scope, a named owner, metrics to watch and a rollback plan;
  • REWORK: specific failure families fail. The fix is scoped, then the same contract replays in full — including the cases that were passing;
  • NO-GO: a blocker is structurally unmet, or failures resist classification. No rollout; the scope or the architecture gets rethought.

These three labels are internal to your organisation. They carry no regulatory value and certify nothing. They bound a rollout decision — which is all they are asked to do, and already a lot.

Turn real failures into regression tests

Once the agent runs in a limited rollout, every incident that comes back is a test candidate: reproduce it as a fictional case, add it to the library, attach a criterion. The NIST playbook asks precisely for documented test sets and pre/post-deployment comparison on that basis.[3]

That loop is what makes an eval suite durable: the case library grows with observed reality, and every future version is judged on a corpus that contains its own past failures. An agent that no longer regresses on its known incidents is the only honest operational meaning of “reliable”.

The acceptance checklist: what to ask a vendor before a pilot

If a vendor pitches you an agent, ask to see — and treat “we tested it, it works” as a non-answer:

  1. the eval contract: tasks, inputs, expected end states, forbidden actions;
  2. the case library: normal, ambiguous, incomplete, out-of-scope, tool-failure cases;
  3. the release blockers and the grader family attached to each;
  4. an annotated trace: one complete run, readable by a non-engineer;
  5. the failure classification by family, not a lone average score;
  6. the baseline vs candidate comparison at explicit versions;
  7. the decision process: who signs the GO, and what the rollback plan is.

A serious vendor can produce this evidence or build it with you. The one who cannot is showing you a demo.

If your agent prototype works in a demo but still has no acceptance evidence, Last Word can turn the workflow into a practical eval contract: scenarios, forbidden actions, trace checks, release blockers and a limited-rollout decision. Explore our AI and automation services, or send us the context.

FAQ

How do you evaluate an AI agent before production?

Turn the business question into an eval contract: representative cases with expected outcomes, forbidden actions, observable traces and release blockers, ending in a bounded GO / REWORK / NO-GO decision. A demo is not part of that evidence — it shows a possibility, not repeatable behaviour.

What is the difference between a capability test and a regression test?

A capability test measures whether the agent can do something new. A regression test checks that an already-acquired behaviour survived a change of model, prompt or tools.[1] Both replay against the same eval contract, with explicit versions of everything that changed.

Can an LLM-as-a-judge sign off a production release on its own?

No. A model judge is a probabilistic component, useful for qualities that are hard to code such as tone or relevance, but it must be calibrated against human judgement and is neither an independent authority nor sufficient evidence.[1] Release blockers should be verifiable by deterministic rules, and final acceptance stays human.

How many test cases do you need before a pilot?

No honest universal number exists: the right corpus depends on scope, input diversity and risk. The useful criterion is coverage — every family (normal, ambiguous, incomplete, out-of-scope, tool failure, untrusted content) is represented, and every real incident becomes a regression case.

Who should decide whether an AI agent goes to pilot?

A named owner, signing a bounded decision: GO with scope and rollback plan, REWORK with scoped fixes, or NO-GO. The decision is internal and carries no regulatory value — it bounds a rollout, it does not certify the agent’s safety or compliance.

Sources

[1] https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents — Anthropic, “Demystifying evals for AI agents”, published 9 January 2026, accessed 3 September 2026.

[2] https://www.nist.gov/programs-projects/building-evaluation-probes-agentic-ai — NIST, “Building Evaluation Probes into Agentic AI”, published 7 April 2026, accessed 3 September 2026.

[3] https://airc.nist.gov/airmf-resources/playbook/measure — NIST AIRC, AI RMF playbook, Measure function, accessed 3 September 2026.

[4] https://developers.openai.com/api/docs/guides/agent-evals — OpenAI, “Evaluate agent workflows”, no publication date displayed, accessed 3 September 2026. Vendor guidance.

[5] https://learn.microsoft.com/en-us/microsoft-copilot-studio/guidance/evaluation-overview — Microsoft Learn, “Design and operationalize agent evaluation”, no publication date displayed, accessed 3 September 2026. Vendor guidance.