How to evaluate an agent

Why the chatbot playbook fails

Evaluating a chat model is comparatively easy: one input, one output, and a human can read the output and say whether it is any good. Evaluating an agent is a different problem, because the thing you are judging is not an answer. It is a trajectory - twenty decisions, each conditioned on the last, most of them invisible by the time you see the result.

That difference has a practical consequence that catches teams out. An agent can produce exactly the right output through a route you would never accept: three failed attempts against the wrong API, a file deleted along the way, four times the expected spend. Graded on output alone that run is a pass. Graded honestly it is a warning about the next run, which will take the same route and may not get lucky.

So agent evaluation has two halves, and they answer different questions:

Define done outside the agent

Nothing else in this article matters if completion is judged by the model. As Why agents fail argues, an agent asked whether it finished is evaluating a goal it may have partly lost from context, against evidence it selected itself. Every input to that judgement is controlled by the thing being judged.

An evaluable task therefore needs a success condition the harness can check without asking the model anything:

• Existence. The file, the record, the pull request is where it should be.
• Shape. The output validates against a schema, parses as JSON, has the required fields.
• Count. Three invoices in, three drafts out. The cheapest and most under-used check there is.
• Behaviour. The tests pass, the query returns the expected rows, the build succeeds.

Tasks that cannot be reduced to something in that list are not un-evaluable, but they are expensive to evaluate, and they should be a small deliberate fraction of your suite rather than the bulk of it by accident.

Judge the path, not just the answer

Trajectory metrics are cheap to collect - the harness already has the data - and almost nobody looks at them until something goes wrong. The useful ones are ordinary:

• Step count against a baseline. A task that normally takes six steps and took nineteen has told you something even though it passed.
• Tool-call error rate. Rising failed calls with a flat success rate means the model is compensating for a broken tool. That compensation runs out.
• Retries and repeats. The same call with slightly different arguments, more than twice, is a stall the run recovered from by luck.
• Side effects. Writes outside the expected paths, destructive operations, anything touched that the task did not require.
• Cost and wall clock. Both per run and at the 95th percentile, because the tail is what breaks budgets.

The single most valuable derived number is the gap between outcome pass rate and clean-path pass rate. When 95% of runs succeed but only 78% succeed cleanly, that 17-point gap is your leading indicator: those runs are already failing, they just have not been caught out yet.

Outcome tells you what happened. Path tells you what is about to happen.- why one number is never enough

Building a suite that does not lie

An offline suite is a fixed set of tasks with known-good outcomes, run against a controlled environment. It is the backbone of agent evaluation and it is very easy to build one that reports numbers with no relationship to production.

The properties that matter:

• A seeded, resettable environment. If the suite mutates state, every run after the first is measuring a different task. Snapshot and restore between runs.
• Real tasks, including the ugly ones. Suites drift toward the tasks that were easy to write down. Deliberately include the ambiguous request, the missing file, the tool that returns an error, the task that should be refused.
• Cheap enough to run often. A suite that takes four hours and $200 gets run before releases and then stops getting run. Forty well-chosen tasks beat four hundred.
• Held-out tasks. Keep a portion the prompt authors never see. Prompts get tuned against whatever is visible - that is not cheating, it is gradient descent by human - and without a held-out set you cannot tell tuning from improvement.

Two failure modes to watch for in the suite itself. Overfitting: the numbers climb while production does not, because the suite has become the specification. Staleness: the suite tests last quarter's task mix, so it passes cleanly while users hit cases it never covered.

Probes on live traffic

Offline suites answer "did this change break something we already knew about". Only production answers "is this working now". The three techniques that pay for themselves:

• Structural checks on every run. The same mechanical assertions from section 02, applied in production, not just in the suite. They cost almost nothing and they catch the silent partial success that no sampling scheme will reach in time.
• Model-as-judge on a sample. A second model scoring completeness and grounding is genuinely useful for triage. It is not a measurement instrument: it drifts with the model, it is generous about plausible-looking work, and it shares blind spots with the agent it grades. Use it to decide what a human should look at, not to publish a quality number.
• Human review of successes. The counterintuitive one, and the one that works. Failures are already visible. Pull ten runs a week that the system called successful and check them properly. This is how most teams discover their real pass rate is not the one on the dashboard.

Catching drift

Agents rarely break. They decline. A model upgrade, a prompt tweak, a tool whose output format changed slightly, a shift in what users are asking for - each takes a few points off, none trips an alarm, and six weeks later the system is meaningfully worse with no incident to point at.

Detecting that needs a small number of metrics tracked continuously rather than a large number checked occasionally:

• Outcome pass rate and clean-path pass rate, plotted together - the gap between them is the signal.
• Median and p95 steps per run.
• Cost per successful run, which absorbs both retries and context growth.
• Escalation rate - how often a run ends up with a human.

Two disciplines make those numbers trustworthy. Pin the version of everything - model, prompt, tool schema - so a change in the graph maps to a change you made. And re-run the suite on the old version when you suspect drift, because a moving baseline explains more surprising regressions than any bug does.

Acting on the numbers

Evaluation earns nothing until it changes a decision. Three that are worth wiring up:

• Gate releases on the suite. A prompt or model change that drops the held-out pass rate does not ship, the same way a failing test does not merge.
• Feed production failures back as test cases. Every real failure becomes a task in the suite. This is the loop that makes the suite representative over time instead of drifting away from reality.
• Let the numbers set autonomy. Trust should be earned against measurements, not against a feeling that it has been working lately. A task type with a high clean-path rate over months is a candidate for fewer approval gates; one without it is not, however good the demo looked.

And know when an eval is lying. A suspiciously high pass rate usually means the suite is too easy or the checks are too loose. A number that has not moved in months usually means nobody is running it. Both are more common than a genuinely stable agent.