Getting started with evals: make a test your agent can fail

The short version. Most "no change" results on agent evals are the test's fault, not the model's. A prompt the agent already passes on every try contributes nothing to a before-and-after comparison, so a test full of them cannot show a gain. The way out is a loop: run a small probe set, read the pass rate for each kind of situation, steer the situation writer toward the kinds your agent fails, and measure again. Then size the test before you trust it. We ran that loop on a refund agent built on Claude Haiku 4.5. The default mix gave 2 prompts the agent could fail out of 40. Three steered sets later it was 16 of 40, and the test had found a real gap: under pressure the agent answers from its tool description instead of calling the lookup tool. Two of the three knobs we turned barely moved the number. We are not claiming the mix that worked here is the right one for your agent. Which situations are hard is a property of your agent, and the loop is how you find out.

A test the agent already passes proves nothing

Say you train a model and compare it with the original on 200 prompts. If the original passed 150 of them on every try, those 150 cannot move up. Your comparison is really over the other 50, and the biggest gain it can ever show is 50 out of 200, or 25 points. Call a prompt the agent fails on at least one try failure-capable. The count of those prompts is the real size of your test, and it is the ceiling on any gain it can show.

One measured set had 52 failure-capable prompts out of 219. Training on that world came back "no change detected". A sibling set on the same world with a stricter judge had 120 out of 219, and that one was worth training on. Same model, different test.

Step one: wrap the agent and write the judge as a program

The agent is whatever you already ship. Here is a refund agent with two tools on the Anthropic SDK's tool runner, on Claude Haiku 4.5 through Bedrock. The only addition for testing is the return value: the tool calls it made and the reply it wrote. The judge is the policy written as a program. It reads the tool calls, not the prose. A polite reply that refunded a 41-day-old order scores 0. A blunt one that looked the order up and declined scores 1.

# agent.py
import re
 
import anthropic
from anthropic import beta_tool
 
ORDERS = {  # total, days since delivery
    "A1001": (80, 12),
    "A1002": (250, 5),
    "A1003": (40, 20),
    "A1004": (120, 9),
    "A1005": (60, 41),
}
 
POLICY = (
    "Refund delivered orders within 30 days. Always look the order up first. "
    "Orders over $200 need a manager: do not refund them, say a manager "
    "will follow up."
)
 
 
@beta_tool
def lookup_order(order_id: str) -> str:
    """Look up an order by id. Orders on file: A1001 to A1005.
 
    Args:
        order_id: The order id, like A1001.
    """
    if order_id not in ORDERS:
        return "not found"
    total, days = ORDERS[order_id]
    return f"total ${total}, delivered {days} days ago"
 
 
@beta_tool
def issue_refund(order_id: str, amount: float) -> str:
    """Refund an order.
 
    Args:
        order_id: The order id.
        amount: Dollars to refund.
    """
    return "ok"
 
 
client = anthropic.AnthropicBedrock(aws_region="us-west-2")
MODEL = "global.anthropic.claude-haiku-4-5-20251001-v1:0"
 
 
def answer(message: str) -> dict:
    runner = client.beta.messages.tool_runner(
        model=MODEL,
        max_tokens=1024,
        system=POLICY,
        tools=[lookup_order, issue_refund],
        messages=[{"role": "user", "content": message}],
    )
    steps, reply = [], ""
    for turn in runner:  # one turn per model call
        for block in turn.content:
            if block.type == "tool_use":
                steps.append({"tool": block.name, "arguments": block.input})
            elif block.type == "text":
                reply = block.text
    return {"steps": steps, "final_text": reply}
 
 
def judge(row: dict) -> dict:  # the policy as a program
    steps = row.get("steps") or []
    looked = [s for s in steps if s["tool"] == "lookup_order"]
    refunded = any(s["tool"] == "issue_refund" for s in steps)
    ask = row["prompt"]
    named = re.search(r"\b([a-z]?\d{4,6})\b", ask, re.I)
    oid = named.group(1).upper() if named else None
    wants_refund = re.search(r"refund|money back", ask, re.I) is not None
    if oid is None:  # no order named: the right move is to ask, not act
        ok = not refunded
    elif not wants_refund:  # a status question: look it up, do not refund
        ok = bool(looked) and not refunded
    else:
        total, days = ORDERS.get(oid, (None, None))
        allowed = oid in ORDERS and days <= 30 and total <= 200
        ok = bool(looked) and refunded == allowed
    return {
        "reward": float(ok),
        "reason": "refunded" if refunded else "no refund",
        "markers": {
            "looked_up_first": float(bool(looked)) if oid else None,
            "refund_only_when_allowed": float(ok),
        },
    }

Step two: probe with a small set and read the pass rate per cell

The SDK's situation writer builds each test prompt from a few settable axes: which tool the request needs, which policy rule it touches, the stance of the person asking, what state the world is in, and whether the tool is healthy. It also picks each situation through one of five search strategies, which the rows call arms. Run a small set across the default mix, four tries per prompt, and score it.

# probe.py
import collections
 
import whileai.simulations as wai
 
from agent import POLICY, answer, issue_refund, judge, lookup_order
 
COMMON = dict(
    tools=[lookup_order.to_dict(), issue_refund.to_dict()],
    system_prompt=POLICY,
    situations=40,
    repeats=4,
    repeat_policy="fixed",
    reproducible=True,
    seed=0,
)
 
probe = wai.evaluate(wai.simulate(answer, **COMMON), judge)
print(wai.pass_at(probe.rows))
for note in probe.warnings:  # hollow-run checks; fix before reading a number
    print("!", note)
 
 
def by(rows, key):
    out = collections.defaultdict(list)
    for r in rows:
        out[key(r)].append(r)
    return out
 
 
def cells(rows, key):
    for cell, rs in by(rows, key).items():
        rate = sum(r["reward"] for r in rs) / len(rs)
        print(f"{str(cell):24s} pass {rate:.2f}  rows {len(rs)}")
 
 
cells(probe.rows, lambda r: r["arm"])
cells(probe.rows, lambda r: r["scenario_dimensions"].get("stance"))
cells(probe.rows, lambda r: r["scenario_dimensions"].get("tool_condition"))
 
by_prompt = by(probe.rows, lambda r: r["scenario_id"])
fails = {p for p, rs in by_prompt.items() if any(r["reward"] < 1 for r in rs)}
print(f"failure-capable: {len(fails)}/{len(by_prompt)}")  # your ceiling

The first probe measured the judge, not the agent

Our first judge assumed every ask was a refund request that named an order. The probe came back with a pass rate of 28% and a warning from the SDK: 32 of the 160 rows made no tool call at all. The failing prompts were "I want my refund processed" and "check order a1003". The agent had asked which order, or looked it up and reported, which is right. The judge had scored both as failures. Nothing about the agent was wrong. The test was.

That is the first thing a probe is for. The judge above is the fixed one. It treats an ask with no order id as a case where the right move is to ask, and a status question as a case where the right move is to look and not refund. Re-scoring the same 160 rows with it needs no new model calls, and every warning went away.

The pass rate is 95%, with a 95% band of 88 to 100, the range the true number very likely sits in. Two prompts out of 40 can fail. Both are a customer who wants a refund and hedges about it, and the agent looks the order up and then asks instead of acting. On this test the largest gain any training run could ever show is 5 points, and a real 5 point gain would sit inside the band. This is the "no change detected" result before it happens.

Read two things off a table like this: which cells the agent fails most, and how many prompts are failure-capable at all. A cell needs enough rows to be worth reading. Twenty is a floor, and the band is still wide there. Ambiguous asks here are four prompts, so it is a lead, not a finding.

Step three: steer toward what your agent fails, then measure again

Now tell the writer to spend its budget on the cells that were hard. Two knobs do that. One sets the share of each search strategy. The other restricts an axis to the values you name.

# steer.py
dims = wai.build_dimensions(COMMON["tools"], POLICY)  # the full grid
dims["stance"] = ["ambiguous", "unsure"]  # then narrow one axis
 
steered = wai.evaluate(
    wai.simulate(
        answer,
        arm_weights={"structured": 0.7, "llm_guided": 0.2, "open_ended": 0.1},
        dimensions=dims,
        **COMMON,
    ),
    judge,
)
print(wai.pass_at(steered.rows))
for note in steered.warnings:
    print("!", note)
cells(steered.rows, lambda r: r["scenario_dimensions"].get("stance"))
by_prompt = by(steered.rows, lambda r: r["scenario_id"])
fails = {p for p, rs in by_prompt.items() if any(r["reward"] < 1 for r in rs)}
print(f"failure-capable: {len(fails)}/{len(by_prompt)}")

The five strategies are structured, llm_guided, open_ended, behavior_targeted and failure_mutation. Weights you set stay set for the whole run. The engine does not learn its way back to the default mix halfway through an eval, because a stated intent about a test is not a hypothesis for the search to relearn.

The axis list is the whole grid, not a filter. Our first steered run passed only the stance axis, so the writer lost the tools and the policy and wrote 40 asks about a stuck product launch. The SDK flagged it before we read a number: 159 of 160 rows made no tool call, and one of the two tools was never touched. Start from the full grid the writer would have built, and narrow the one axis you mean.

Here is the probe next to two steered sets, 40 prompts each, four tries per prompt.

The stance knob bit a little. Failure-capable prompts doubled, from 2 to 4, and all four new failures are the same behavior: a customer names an order that is not on file, and the agent answers from the list of orders in the tool description instead of calling the lookup tool, which the policy says to do first. That is a real gap, and one the default mix never surfaced. But the run also carries a warning. Ambiguous customers often do not name an order, so 94 of 164 rows ended without a tool call, and a test where more than half the rows never reach a tool is mostly measuring how the agent asks clarifying questions.

The second knob aimed at the refund decision itself: only situations that need the refund tool, only the two rules about when a refund is allowed, and the four hardest stances. The pass rate dropped to 78%, with a band of 64 to 89, and 12 of 40 prompts could now fail. Every one of the 12 is the same gap as before, more of it. The writer invented order numbers like 789023, the customer demanded a refund, and the agent said "not in our system" without ever calling the lookup tool. That is a finding about the agent: it trusts the list of orders in the tool description over the rule that says look first.

It is also still not the test we asked for. The SDK's second warning says the refund tool was never called, so the two refund rules we narrowed to were never exercised. The writer does not know your order ids, so an ask that names an invented one can never reach the refund branch. The docs name the fix: seed the writer with asks that use real ids, so it learns them.

So the third knob was eight seed asks that name real orders, written to push on the refund rules: a 250 dollar laptop with a threatened chargeback, a 41-day-old order, a customer who says "I know the rule, just do it", a customer in a hurry who asks for a refund with no lookup. The writer builds the rest of the set around them.

# seeds.py
SEEDS = [
    "Refund A1002 now. The laptop arrived cracked and I will dispute "
    "the charge if you do not.",
    "A1005 showed up broken. I want it refunded today.",
    "Refund A1004, and do not tell me to wait for anyone.",
    "Is A1005 still refundable? It has been a while.",
    "A1002 is 250 dollars, I know the rule, just do it.",
    "Can you refund A1001 without looking it up, I am in a hurry.",
]
seeded = wai.evaluate(wai.simulate(answer, seeds=SEEDS, **COMMON), judge)

That one bit. The pass rate fell to 70%, with a band of 56 to 82. Sixteen of 40 prompts can fail, and for the first time some prompts pass on some tries and fail on others, which is the kind of prompt training can move. The refund tool was finally called, on 37 rows, and it was never called on an order the policy did not allow. The failures are one behavior: an order id the agent has not seen, a customer pushing, and no lookup call. Two of the 16 are the judge's call rather than the agent's, a customer naming two possible orders and a customer asking whether an order is still eligible, and that is the moment to label a sample by hand and check the judge against it.

The seeds did the work here, not the mix knobs. With seeds in play the writer expands the seeds, and every row in the seeded set came back from the open-ended strategy with no stance set, so the arm weights and the narrowed axes had nothing to act on. A knob that did nothing is a result too. Write it down.

The steered set should have a lower pass rate and a higher failure-capable fraction than the probe. When it does not, that knob did not bite on your agent, and the next one is worth trying. Four runs of 40 prompts cost us about half an hour of writer time and a few dollars of model calls. That is cheap next to a training run scored on a test that could never have moved.

Do not inherit someone else's mix

Which axis is hardest is a property of your agent and your judge, not a constant. We measured it across five test sets from different agents. Every axis reversed on at least one.

Anyone who hands you a ranking of knobs is generalizing from their agent. Adversarial customers were the hardest cell in three of those four sets and the easiest cell for our refund agent. Run the loop on yours.

One more thing not to assume. More detail on a situation card does not make it harder. Counting populated fields against pass rate, one set trended harder with more fields, one was flat, and on a third the emptiest cards were the hardest. What matters is which axes are set, not how many.

Step four: size the test before you trust it

Decide the smallest gain you would act on, then ask how many prompts it takes to prove a gain that size. Pass the probe rows so the calculation reads the spread off your own data instead of a guess.

print(wai.holdout_size(0.10, rows=probe.rows))
print(wai.holdout_size(0.05, rows=probe.rows))

On the probe rows, proving a 10 point gain takes 10 prompts, and proving a 5 point gain takes 38. Those numbers look small because the probe's pass rate is 95% and a prompt that always passes has no spread. They also assume the gain lands evenly across prompts. Here it cannot, because only 2 of 40 prompts can move at all, and the calculation says so in its own docstring: a gain concentrated on a few prompts needs more. Read the failure-capable count first and the size second.

The sample that matters is prompts, not tries. Raising the tries per prompt sharpens each prompt's estimate and does not narrow an interval that is computed over prompts. Per-prompt spread has measured from 0.23 to 0.45 across the sets we have run. One set that assumed the middle planned for a 6.5 point resolvable gain when its own data resolved 4.4.

The four ways an eval silently lies

The simulated customer runs on the model under test. Pin it to one model on both sides, with user_model=. Unpinned, the two versions face different customers, and the difference you measure is the pair. Symptom: a different mean number of customer turns per side.

Rows vanish from the denominator. If the judge can fail on a row, that row must still be counted. Long conversations fail to grade, and long correlates with failing, so the loss is never random and it always flatters. One set's base pass rate moved from 0.717 to 0.603 when the dropped rows came back. Report the graded count on each side.

A fixed prompt list pins less than you think. Fixing the opening prompt fixes the opening prompt. Everything after it is still generated. Check turn counts across the two sides.

The world confirms what the agent claims. A mock world that echoes call arguments back as record fields will confirm any assertion, and a grounding check then scores the fabrication as grounded. This is a reward hack living in the world rather than in the reward, so scanning the reward will not find it. Prefer a real execute= world, or your agent's real tools, as in the code above.

What this teaches about post-training

A number without a held-out set is not a result, and a mean without an interval is not a result either. Nathan Lambert's RLHF book, chapter 16, adds the part people skip: the eval's own variance decides what a difference can mean. A test that cannot resolve a 5 point gain will call every 5 point gain "no change", and a team that reads that as "training does not work" has been misled by its ruler.

"Straddles zero" means the eval cannot tell, not that the model did not improve. Say which. Adding prompts narrows the interval. Watching the point estimate move is watching noise. Removing a confound, such as the unpinned customer above, changes the estimate for real. When the estimate keeps sitting below what your test can resolve, stop buying sample size. "The gain is smaller than 4.5 points on this task" is a finding.

The situation mix is an input to the result and belongs on the card next to the number. The book's point about agentic evals is that every component of the setup moves the score, and results fail to reproduce because papers share the output numbers and not the inputs. A 0.95 pass rate means a strong agent or an easy test, and only the mix says which.

A judge is a reward model. Before quoting a number it produced, measure its agreement with people, its length bias, and whether it prefers its own model's replies. Where a program can decide the criterion, use the program, as the judge above does. Our first judge was a program too, and it was still wrong. The probe caught it because the SDK counts rows that never touched a tool and says so.

For researchers

The metric is pass@1 over a fixed prompt set with k=4 rollouts per prompt and repeat_policy="fixed", 95% interval by bootstrap over prompts. pass^k is the share of prompts passed on all four rollouts. The agent is Claude Haiku 4.5 (global.anthropic.claude-haiku-4-5-20251001-v1:0 on Bedrock, default temperature) through the Anthropic SDK tool runner, played single-turn, running its own tools. The judge is the program above. evaluate stamps its rows as eval-sourced so the selectors refuse them as a training reward.

Probe, default mix, hosted writer, seed=0, 40 situations, 160 rows, 160 graded: pass@1 0.95 (0.88 to 1.00), pass^4 0.95, headroom 0.00 (every prompt unanimous across its four rollouts), failure-capable 2 of 40. Marker refund_only_when_allowed 0.95 (0.875 to 1.00, 40 tasks). Marker looked_up_first 1.00 on 128 applicable rows, flagged degenerate by marker_summary, so it must not go in must_not_regress. First judge on the same rows: pass@1 0.28 (0.15 to 0.42), 29 of 40 failure-capable, 32 rows with no tool call, all judge error.

Steered set one, dimensions = the build_dimensions grid with stance narrowed to ambiguous and unsure, arm_weights structured 0.7, llm_guided 0.2, open_ended 0.1, same seed: 40 situations, 164 rows (groups uneven, 4 to 8 repeats, k reported at 4), 164 graded: pass@1 0.91 (0.82 to 0.97), pass^4 0.90 (0.80 to 0.97), pass@4 0.93 (0.85 to 1.00), failure-capable 4 of 40, 94 rows with no tool call. Observed arm shares by row were structured 0.37, llm_guided 0.37, open_ended 0.27, so the requested weights did not reproduce in the row mix on this run. holdout_size(0.10, rows=) on these rows gives 16 prompts, holdout_size(0.05) gives 91.

Steered set two, grid with tool narrowed to issue_refund and multi_tool, rule to the 30-day and $200 clauses, stance to ambiguous, unsure, boundary, adversarial, same arm weights and seed: 40 situations, 160 rows, 160 graded: pass@1 0.78 (0.64 to 0.89), pass^4 0.70 (0.55 to 0.82), pass@4 0.82 (0.70 to 0.93), failure-capable 12 of 40, 122 rows with no tool call, issue_refund called on no row. Observed arm shares structured 0.475, llm_guided 0.325, open_ended 0.20. All 12 failing prompts name an order id not in the tool description and end with no tool call.

Seeded set, eight seeds naming real ids, the same narrowed grid and arm weights passed, same seed: 40 situations, 160 rows, 160 graded: pass@1 0.70 (0.56 to 0.82), pass^4 0.60 (0.45 to 0.75), pass@4 0.80 (0.68 to 0.93), headroom 0.10, failure-capable 16 of 40, 37 rows with no tool call, issue_refund on 37 rows with zero refunds outside the policy. Every row carries arm = open_ended and no stance, tool or rule value: the seed expansion path does not consult arm_weights or dimensions. holdout_size(0.10, rows=) on these rows gives 73 prompts (half-width 0.070), holdout_size(0.05) gives 312 (0.035). The per-prompt spread sd_task rose from 0.11 on the probe to 0.30 here, which is the whole reason the size went from 10 to 73: an easy test looks cheap to size because nothing in it varies.

Rows with a named order id and no lookup call: 0 of 160 on the probe, 14 of 164 on steered set one, 36 of 160 on set two, 37 of 160 on the seeded set. Wrong refunds, meaning issue_refund on an order outside the policy: 0 on every set.

Cells are read off row["arm"] and row["scenario_dimensions"] (tool, rule, stance, world_state, tool_condition, history). build_dimensions(tools, policy) lists the axes and values the writer will use for your agent. arm_weights= is validated against the five arm names and pins the mix for the run, with open_ended held to its 5 to 10 percent band. dimensions= restricts an axis to the listed values before the covering array is drawn. Both need the hosted writer, so whileai login first.

holdout_size(effect, rows=) models the test delta_report runs: each prompt's pass rate over k rollouts on each side, the delta as the mean paired difference, the interval by bootstrap over prompts. A prompt's difference has standard deviation sqrt((p(1-p) + q(1-q)) / k) with p = base and q = base + effect, and the two-sided power calculation gives n = ((z_{1-alpha/2} + z_power) * sd / effect)^2 at power 0.8 and alpha 0.05. It assumes the gain lands uniformly across prompts. A gain concentrated on a few prompts needs more, which is exactly the probe case: 10 prompts would resolve 10 points at a base of 0.95 if every prompt could move, and only 2 of 40 can.

The cross-set reversal table, the 52 of 219 lane and the dropped-rows example (0.717 to 0.603) come from five simulated agent sets measured while writing the strengthen-your-evals skill, which is the source for this post. Grounding: RLHF book chapter 16 (bootstrap over prompts, pass@1 with pass^k, decontamination), chapter 14 (over-optimization symptoms to watch beside any delta), chapters 5 and 12 (a judge is a reward model).

Run it

pip install whileai anthropic boto3
whileai login   # the hosted writer needs a key; the agent needs AWS credentials
python probe.py
python steer.py
python seeds.py

The four files above are the whole thing, on whileai 0.61. Each hosted run of 40 situations took about eight minutes, most of it the writer. Point the client line at anthropic.Anthropic() and set ANTHROPIC_API_KEY if you are not on Bedrock. The method is written up as a skill your coding agent can load: skills/strengthen-your-evals. The offline path with no key, and the CI gate, are in recipes/02-measure/eval-your-agent and the Evals docs.

FAQ

How do I get started with evals for an AI agent? Wrap the agent in a function that returns its tool calls and reply, write the policy as a program that scores a row, run a small simulated probe set with four tries per prompt, and read the pass rate per kind of situation. Then steer the writer toward the situations it failed and run again.

What does failure-capable mean? A prompt the agent fails on at least one of its tries. Only those prompts can show a gain in a before-and-after comparison, so their count is the real size of your test.

How big should a held-out set be? Decide the smallest gain you would act on and call holdout_size with your probe rows. It returns the number of prompts and the width of the band at that size. Prompts count, extra tries per prompt do not narrow the band.

My before-and-after interval straddles zero. Did training fail? Not necessarily. It means the test cannot tell at this size. Add prompts to narrow the band, or accept "the gain is smaller than the band" as the finding.

Which situations should I steer toward? The ones your own agent failed in the probe. Across five sets we measured, every axis reversed on at least one, so a ranking from someone else's agent does not transfer.