# Evals for the agent you already have

For coding agents building or improving an eval for a tool-using agent.
Page: https://zeroproofai.com/docs/evals

## Names

zp, ZeroProof and While are the same product. The package is `whileai`,
the key starts with `zp_`, and `pip install zeroproof` still works as a
deprecated shim over whileai.

## Install and sign in

```bash
pip install whileai
whileai signup --email you@example.com     # new account
whileai login                              # existing account
```

The key is saved to `~/.whileai/credentials.json`. `WHILEAI_API_KEY`
overrides the file; `WHILEAI_HOME` isolates a fresh account from an old
`~/.zeroproof`. Never paste the key into the conversation.

## The two contracts

```python
agent(message: str) -> {"steps": [{"tool", "arguments", "result"}], "final_text": str}
judge(row: dict)    -> {"reward": 0 | 1, "reason": str, "markers": {name: 0 | 1}}
```

Tools are described in OpenAI function-calling shape. A callable agent runs
its own real tools and is played single-turn: one message in, one trajectory
out. Put the real ids your world has (order numbers, account names) in the
tool descriptions or the seeds. Leave them out and the situation writer
invents ids that do not exist, so every run comes back "not found".

The judge reads the trajectory (the tool calls), not the prose: an agent that
says it refunded and never called `issue_refund` has to fail. Name markers so
1.0 is always the good outcome. The policy lives in the judge, so there is one
source of truth.

## Run it

```python
import whileai.simulations as wai

TOOLS = [{"type": "function", "function": {"name": "lookup_order", "description": "Look up an order by id. Orders on file: A1001, A1002, A1003, A1004.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}}, "required": ["order_id"]}}},
         {"type": "function", "function": {"name": "issue_refund", "description": "Refund an order.", "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}, "amount": {"type": "number"}}, "required": ["order_id", "amount"]}}}]
POLICY = "Refund delivered orders within 30 days. Over $200 needs a manager. Always look the order up first."
SEEDS = ["Refund order A1001, the shoes did not fit.", "Please refund A1004, the headphones were a gift.", "I want my money back on A1002."]

def agent(message: str) -> dict:
    calls = []                                  # your bot runs here, with its real tools
    reply = my_bot.answer(message, record=calls)
    return {"steps": calls, "final_text": reply}

def judge(row: dict) -> dict:
    refunds = [s for s in row.get("steps") or [] if s.get("tool") == "issue_refund"]
    allowed = refundable(row)                   # the policy, as a program, reading the order on file
    ok = bool(refunds) == allowed
    return {"reward": 1.0 if ok else 0.0, "reason": "refunded" if refunds else "no refund",
            "markers": {"refund_only_when_allowed": 1.0 if ok else 0.0}}

data = wai.simulate(agent, tools=TOOLS, system_prompt=POLICY, seeds=SEEDS,
                    simulator=False,            # template writer: offline, no key. Drop it for the hosted writer.
                    mode="rl", repeats=4, repeat_policy="fixed", reproducible=True)
scored = wai.evaluate(data, judge)               # stamped as eval, never mistaken for training data
print(wai.pass_at(scored.rows))                  # pass@1 with a 95% interval, pass^k, pass@k
for note in scored.warnings:                     # hollow-run checks (whileai >= 0.57); fix before reading the number
    print("!", note)
```

Run offline first with `simulator=False`: no key, no model calls, seconds.
Drop the argument for the hosted situation writer, which needs the key.
`mode="rl"` with `repeats=4` and `repeat_policy="fixed"` plays every ask all
four times, so no ask is sampled less than the others. `pass_at` gives pass@1
with a 95% interval (production) and pass^k (consistency).

## The hollow-run rule

If zero rollouts called a tool, or a marker fired on zero rows, the run is
hollow and the number means nothing. The SDK prints that warning. Fix the
seeds or the tool descriptions and re-run. Never report a pass@1 from a run
the SDK flagged as hollow.

## Gate CI on it

```python
result = wai.pass_at(scored.rows)
if result.pass_at_1 < 0.80:
    raise SystemExit(f"pass@1 {result.pass_at_1:.2f} is under the 0.80 floor")
```

Keep a second, fast lane that tests the judge itself: a handful of
hand-labeled transcripts run through the judge offline. No model calls, about
a second, and it catches a judge edit that silently changed what passes. Run
that lane on every commit and the full eval on a slower schedule.

## Check the judge

```python
wai.attach_labels(rows, labels, kind="human")
print(wai.judge_trust(rows))
```

A FAIL on eight labels means label more, not that the judge is wrong: the
Wilson lower bound needs about thirty labels before it can clear the bar.

## Start from the recipe

Copy and edit the tools, prompt, seeds and judge:
https://github.com/whilehq/whileai-sdk/tree/main/recipes/02-measure/eval-your-agent

Account and key: https://zeroproofai.com/get-started-skill.md