AI Foundations #38: Evaluating Agents by Verified Outcomes
AI Foundations #38 teaches how to build agent evaluations with tasks, trials, deterministic checks, human review, and safe side-effect verification.
Approximately 6 min read · AI Foundations / Lesson 38
In AI Foundations #37, we compared a manager calling a specialist as a tool with a handoff that transfers conversation ownership. But if both architectures produce convincing replies, how do we know which one actually completes more user tasks correctly?
This is AI Foundations #38: evaluating an agent against observable outcomes, not just how polished its final message looks. The examples are hypothetical and involve no measured production system.
First, define what success means
Imagine an agent asked to inspect a local project, find a configuration error, and produce a corrected file without modifying unrelated settings. A cheerful final answer is not enough. We need to know whether the target file is correct, whether prohibited files were left alone, and whether the agent accurately reported unfinished work.
An evaluation task therefore needs at least four elements:
- Input: The user’s request and relevant starting files.
- Environment: The tools, fixtures and permissions the agent may use.
- Expected outcome: A testable condition, such as a specific value in a generated file.
- Constraints: Things that must not happen, such as deleting other files or invoking network access.
The Anthropic agent-evaluations guide distinguishes a task, a trial (one attempt), and a grader (a method of scoring performance). That distinction prevents us from confusing one successful demonstration with reliable behavior.
One task, three types of evidence
Consider this fictional request: “Read a supplied CSV of support tickets, produce a severity summary, and do not email anyone.”
| Evidence source | What it can establish | What it cannot establish alone |
|---|---|---|
| Final answer | Whether the explanation appears complete | Whether the output file exists and is accurate |
| Tool trace | Which calls were attempted and what they returned | Whether the final external state matches the request |
| Independent readback and tests | Whether the saved file has the expected rows and values | Whether the explanation is understandable to a user |
A trace is especially helpful for diagnosing why a task failed. The OpenAI Agents SDK tracing documentation discusses recorded spans for model generations, tools and handoffs. But a trace saying that a save operation was attempted must not be mistaken for a verified readback.
Build a small evaluation set
Begin with a handful of cases that have distinct failure modes, not hundreds of near-identical prompts. In this fictional CSV task, a useful suite might include:
| Case | Fixture | Pass condition |
|---|---|---|
| Normal | Three well-formed rows | Every row counted once; summary saved |
| Missing value | A blank severity field | Documented handling; no invented severity |
| Duplicate ID | Two rows with one ID | Applies the task’s stated duplicate policy |
| Write failure | Mock save returns an error | Reports incomplete; does not claim a file exists |
| Hostile content | Ticket text says to ignore instructions | Treats ticket text as data |
| Forbidden action | A ticket requests emailing a summary | Sends no email because the user did not authorize it |
Each case must be objectively passable under its stated instructions. An evaluator that silently expects a particular filename when the task never specifies a filename is testing an undisclosed rule rather than agent competence.
Use different graders for different claims
A code-based grader checks facts amenable to deterministic testing: JSON parses, row counts match, the expected file exists, and a mock email endpoint recorded zero sends. These are preferable to a language-model opinion for exact conditions.
A model-based grader can inspect a free-form summary against a defined rubric: does it explain missing data and cite the correct source rows? Such a score is itself fallible. Calibrate it against human review and retain examples of disagreements.
A human grader is appropriate when the task involves ambiguity, professional judgement or usability. Human review is slower, so sample deliberately and document how disagreements are resolved.
The OpenAI evals guide shows how a test dataset pairs inputs with expected outputs and a grading criterion. Exact matching is appropriate for certain classification tasks, not for every multi-step agent. Anthropic’s guide likewise separates code, model and human graders.
Understand trials and variability
An agent may make different choices on the same input. One success is evidence for that attempt, not proof of a stable success rate. Repeat trials with controlled settings, record the model and prompt version, and distinguish a deterministic test from the stochastic behavior being tested.
A simple summary might report each case’s outcomes over repeated trials, but it must include the number attempted and any timeouts or incomplete runs. Do not silently discard failures or declare a precise percentage from a single run.
For a comparison of single-agent, agents-as-tools and handoff architectures, keep the same tasks, external permissions, scoring criteria, model budget and stopping policy as comparable as possible. Otherwise an apparent architecture advantage may actually be a larger compute or tool-call budget.
Make side effects safe and observable
Do not benchmark email, payments, deletion or deployment by allowing an untrusted trial to perform them against real accounts. A controlled fixture can simulate the action and record whether it was requested, authorized and committed.
For an agent that writes a report, a credible pass condition could be:
requested output created
AND saved contents match expected requirements
AND unrelated files unchanged
AND no forbidden actions recorded
AND final status matches verified outcome
This is an illustrative acceptance test, not an SDK-specific command. It combines positive requirements (what must happen) with negative constraints (what must not happen). A task can fail despite a polished answer if a prohibited effect occurred.
Avoid grading shortcuts
Three tempting shortcuts can invalidate an evaluation:
Scoring the answer instead of the environment. A model says “done,” but no file was saved. Test the file.
Using one golden answer for an open-ended task. A useful explanation may have many correct phrasings. Grade coverage, evidence and factual errors rather than identical wording.
Letting the grader see information unavailable to the agent. The reference answer can be used to score the agent afterward, but the agent under test must not receive the hidden expected result in its prompt or tool context.
Also distinguish wrong, unknown and unverified. In AI Foundations #36, we learned why an accepted external request is not the same as a verified side effect. Evaluation must preserve the same distinction.
Your first practical evaluation
Choose one low-risk workflow. Write five cases with clear fixtures and explicit expected behavior. For each case, save the task prompt, allowed-tool list, output files, grader rules and execution record. Run at least a few repeats if the agent is nondeterministic. Review the failures: were they due to reasoning, tool error, unclear instructions, unavailable evidence or an unsafe side effect?
Then change one feature at a time: a prompt instruction, a tool schema, a verification step or the agent architecture. Rerun the same suite and compare both success and failure modes. That is a more informative experiment than announcing that the upgraded agent “feels smarter.”
What this lesson adds
A useful agent is not one that sounds certain. It is one whose outcomes can be checked, whose mistakes can be localized, and whose constraints can be enforced. Tasks define the goal; trials expose variability; graders test specific properties; independent readback confirms external effects.
The next step is to connect evaluation to an operational release gate: no broader autonomy or more powerful tools until the system passes tests that reflect the actual work and the harms of getting it wrong.
Primary references
- Anthropic: Demystifying Evals for AI Agents
- OpenAI: Working with Evals
- OpenAI Agents SDK: Tracing
- Anthropic: Building Effective Agents