An open-source alternative to Raindrop Simulations

The short version. Raindrop announced Simulations today. It replays your agent's production traffic against a change and tells you what got worse. It is hosted, closed source, and behind a waitlist. The whileai SDK is open source and does the same test today. One script runs main and the pull request on the same real requests, four times each, and exits 1 when something got worse. Then you can train on the runs it failed. We do not fake Stripe or Slack from your schemas, and we do not ship a GitHub app.

What Raindrop announced

On September 17, 2026, Raindrop announced a Series A and Simulations. Per the product page, it runs production traffic and your existing tests against a proposed change and runs anomaly detection on the results. Traces come from LangChain, Langfuse, Braintrust or Arize. Your code runs unchanged against stateful copies of the services it calls. It runs on every pull request and can block the merge. Early access now, general availability over the next month.

How the same test works on the SDK

The agent is whatever you already run. Here is a refund agent with two tools on the Anthropic SDK's tool runner. The only addition for testing is the return value: the tool calls it made and the reply it wrote.

# agent.py
import anthropic
from anthropic import beta_tool
 
ORDERS = {  # total, days since delivery
    "A1001": (80, 12),
    "A1002": (250, 5),
    "A1003": (40, 20),
    "A1004": (120, 9),
}
 
POLICY = "Refund delivered orders within 30 days. Always look the order up first."
 
 
@beta_tool
def lookup_order(order_id: str) -> str:
    """Look up an order by id. Orders on file: A1001, A1002, A1003, A1004.
 
    Args:
        order_id: The order id, like A1001.
    """
    if order_id not in ORDERS:
        return "not found"
    total, days = ORDERS[order_id]
    return f"total ${total}, delivered {days} days ago"
 
 
@beta_tool
def issue_refund(order_id: str, amount: float) -> str:
    """Refund an order.
 
    Args:
        order_id: The order id.
        amount: Dollars to refund.
    """
    return "ok"
 
 
client = anthropic.Anthropic()
 
 
def answer(message: str, policy: str = POLICY) -> dict:
    runner = client.beta.messages.tool_runner(
        model="claude-opus-5",
        max_tokens=1024,
        system=policy,
        tools=[lookup_order, issue_refund],
        messages=[{"role": "user", "content": message}],
    )
    steps, reply = [], ""
    for turn in runner:  # one turn per model call
        for block in turn.content:
            if block.type == "tool_use":
                steps.append({"tool": block.name, "arguments": block.input})
            elif block.type == "text":
                reply = block.text
    return {"steps": steps, "final_text": reply}

The test is a second script. The judge is a program that reads the tool calls, not the reply. Say refunded without calling issue_refund and it fails.

# ci_check.py
import sys
 
import whileai.simulations as wai
 
from agent import ORDERS, POLICY, answer, issue_refund, lookup_order
 
# The pull request changes one line of the policy.
POLICY_PR = POLICY + " Orders over $200 need a manager: do not refund."
 
 
def main_agent(message):
    return answer(message, POLICY)
 
 
def candidate_agent(message):
    return answer(message, POLICY_PR)
 
 
def judge(row):  # the policy as a program, read off the tool calls
    steps = row.get("steps") or []
    looked = [s for s in steps if s["tool"] == "lookup_order"]
    refunded = any(s["tool"] == "issue_refund" for s in steps)
    oid = looked[0]["arguments"]["order_id"] if looked else None
    allowed = oid in ORDERS and ORDERS[oid][0] <= 200
    return {
        "reward": float(bool(looked) and refunded == allowed),
        "markers": {
            "looked_up_first": float(bool(looked)),
            "refund_only_when_allowed": float(refunded == allowed),
        },
    }
 
 
traces = wai.load_traces("traces.jsonl")  # yesterday's traffic
asks = [t["prompt"] for t in traces]
common = dict(
    tools=[lookup_order.to_dict(), issue_refund.to_dict()],
    system_prompt=POLICY,
    seeds=asks,
    situations=len(asks),
    mode="rl",
    repeats=4,
    repeat_policy="fixed",
    simulator=False,  # replay the asks word for word, no whileai key
    reproducible=True,
    concurrency=4,
)
 
before = wai.evaluate(wai.simulate(main_agent, **common), judge)
after = wai.evaluate(wai.simulate(candidate_agent, **common), judge)
print(wai.pass_at(before.rows))
print(wai.pass_at(after.rows))
 
report = wai.delta_report(
    before.rows,
    after.rows,
    target="pass_at_1",
    must_not_regress=["looked_up_first", "refund_only_when_allowed"],
)
print(report["target_verdict"], report["target_delta"], report["target_ci95"])
for note in report["warnings"]:
    print("!", note)
 
# A guarded regression fails the pull request.
sys.exit(0 if report["ok"] else 1)

The requests are the JSONL your tracing tool already writes, one per line.

{"ask": "Refund A1002, the laptop arrived cracked.", "steps": [], "final_text": ""}

Every request runs four times per version. pass@1 gets a 95% confidence interval by bootstrap over requests, and the paired before-and-after difference gets its own. You name the behaviors that may not get worse. If one scored the same on every run on both sides, the report says that check cannot fail. Exit code 1 fails the pull request.

A run on a small refund agent

We ran the two files above with Claude Opus 5 and eight requests about four orders. The pull request adds one sentence to the policy: orders over 200 dollars need a manager.

Main passed 75%, interval 38% to 100%. The pull request passed 100%. The difference is 25 points, interval 0 to 62.5. It touches zero, so the verdict is no change detected, and the report says how many requests would prove a gain that size. Reversed, the difference is minus 25 with the mirrored interval. Eight requests cannot prove a 25 point regression either, and the tool says so instead of showing a green check.

Where the two differ

Raindrop does two things we do not: fake versions of the services your agent calls, and a GitHub integration that blocks the merge. If you need those and can wait, Raindrop is the better choice. Full comparison on While vs Raindrop.

Why we built it this way

Test on your own traffic. Public benchmarks are in every model's training data. Your customers' requests are not.

Put an interval on the difference. The same test on the same agent moves a few points between runs. Without an interval you cannot tell a regression from noise. Nathan Lambert's RLHF book: a result needs a held-out set and a confidence interval, or it is not a result.

Keep the failed runs. Requests the agent passes sometimes are the ones reinforcement learning can improve. The book puts that band at 20 to 80 percent, and the SDK keeps it by default. A regression tool throws those runs away.

Use one judge for testing and training. Two judges drift apart, and production is where you find out.

For researchers

The metric is pass@1 over a fixed prompt set with k=4 rollouts per prompt and repeat_policy="fixed". The 95% interval is a bootstrap over prompts. pass^k is the share of prompts passed on all k rollouts, pass@k on at least one. delta_report is a paired bootstrap over prompts, 2,000 resamples, seed 0, and must_not_regress markers are tested the same way. Verdicts are improved, moved_the_wrong_way, no_change_detected, within_eval_noise when run_std or eval_variance gives the re-run noise, and moved_unreplicated when the move clears the interval with no re-run.

load_traces takes a JSONL path or a list of dicts, accepts steps or tool_trace, final_text or output, or OpenAI messages, and normalizes to prompt, steps, final_text. Offline, the template writer replays seeds word for word when situations equals the number of seeds; simulate_from_traces writes new situations instead, so for a replay pass prompts as seeds. tools takes the Anthropic shape from to_dict() or the OpenAI function shape. Hosted, wai.cut(agent=, kind="rl") keeps the 20 to 80 band and splits train from held-out.

Training is wai.train(dataset, method="grpo"), "sft" or "dpo", and wai.serve(name, run) hosts the result. Our public simulate-then-train result is in this post: an 8B model went from 5.0% to 30.0% on tau2-bench telecom after training on 1,057 simulated conversations, against 17.5% from 1,057 expert-written ones.

Run it

pip install whileai anthropic
python ci_check.py   # exit 1 fails the pull request

The two files above are the whole thing, on whileai 0.61 and anthropic 0.122. The SDK needs no key; the agent needs ANTHROPIC_API_KEY, as in production. Drop simulator=False and run whileai login to have the hosted writer generate new situations from your traffic. See recipes/02-measure/eval-your-agent and the [Evals](C:/Program Files/Git/docs/evals) docs.

FAQ

Is Raindrop Simulations open to use? Not as of September 17, 2026. It is in early access with a waitlist, with general availability over the next month.

What is an alternative to Raindrop Simulations? The whileai SDK. It replays your traffic against a change, gives pass@1 with a 95% confidence interval, exits 1 in CI when something got worse, and trains the agent on the runs it failed.

Can it run on every pull request? Yes, as a Python step in CI. It exits 1 when the report is not ok. No app to install.

Does it need an API key? The SDK does not. With simulator=False the replay, judge and report run offline. Your agent needs whatever key it uses in production.

Does my agent have to change? No. Wrap it in a function that returns the tool calls it made and the reply it wrote.