Workflow · platform

Agent, datasets, profile, publish

Everything on the platform hangs off an agent. You push sets with a purpose, read the profile before you train, publish the ones worth sharing, and purge the rest. Six calls.

Download skill (.md)

1. Sign in

bash

pip install zeroproof
zeroproof login                            # existing account, one click
zeroproof signup --email you@example.com   # new account, no browser

Details and the trial limits are on get started.

An agent is a record on your account. It exists the moment a push names it or a trace arrives with gen_ai.agent.name. Traces, datasets, evals and public cards all hang off it, and the agent row at the top of Traces and Datasets lists every one you have.

python

import zeroproof.simulations as zps

zps.register_agent("airline-support", description="Refunds and rebooking")   # optional
zps.agents()   # every agent: traces, sets by purpose, public cards

3. Push train and holdout

Every dataset has a purpose: train, holdout or eval. A push is train unless it says otherwise. Ingested traces are eval until you cut training data from them. There is no unassigned state.

python

data = zps.simulate(spec="specs/airline", mode="rl", budget=200)
data.grade()

entry = data.push("airline-v3", agent="airline-support", holdout=0.2)
entry["datasetId"]              # the train set
entry["holdout"]["datasetId"]   # the holdout set, split by task, linked to it

data.push("airline-evals", agent="airline-support", purpose="eval")
zps.update_dataset("ds_...", purpose="holdout")   # move one later

Holdout

Split by scenario_id, so a task is wholly on one side and lands on the same side every run. That is what makes a before and after honest.

Mode

The simulation mode (explore, sft, rl, adaptive) is recorded on the set.

Gate

push runs the publish gate first: an RL-shaped set that is ungraded or has no mixed group is refused. gate=False uploads as is.

4. Read the profile before you train

One pass over the rows, cached until the set changes. The Datasets page shows it on every card; the SDK returns it.

python

p = zps.profile("ds_...")
p["pass_rate"], p["support"], p["mixed_tasks"], p["tasks_with_repeats"]
p["per_task"]        # top tasks with rows, graded, pass rate
p["tool_calls"]      # mean per row, max, top tools

pass rate: rows with reward ≥ 0.5 over graded rows. Near 0 or near 1 means no headroom.
support: mean p(1−p) / 0.25 over tasks with two or more graded rows. The gradient a group gives. 0.3 and up trains; under 0.05 is flat.
mixed tasks: tasks with both a pass and a fail. RL needs these; SFT needs passes.
rows per task: repeats. Below 2 there is no group and no support.
repeated prompts: rollouts that share a prompt. Expected for RL, a warning for SFT.

5. Publish the ones worth sharing

python

zps.publish("ds_...", agent="airline-support", description="Graded refund conversations.")
zps.catalog()                # every public card, grouped by agent
rows = zps.pull("ds_...")    # anyone, no key
zps.unpublish("ds_...")

Cards live at /datasets with rows, size, support, pass rate and three sample rows. A weak set looks weak there on purpose.

6. Evals come from traces

An eval run is what your agent already produces: one trace per case, tagged with the set it belongs to. Send traces with OpenTelemetry, and the Eval section of Datasets shows every set with its pass rate and the change since the last run. Nothing to upload.

7. Purge what you do not want

bash

zeroproof purge --agent demo-agent --dry-run   # count its traces, datasets, record
zeroproof purge --agent demo-agent             # delete them, after a y/N
zeroproof purge --empty --max-rows 2           # datasets with no bytes, or 2 rows or fewer

Python: zps.purge_agent("demo-agent"), zps.delete_empty_datasets(max_rows=2). Both take dry_run=True. Deletion is permanent.

Without Python

Every call above is one route on the gate, key as X-Api-Key.

http

GET  /agents                         POST /agents {name, description?, tools?, system_prompt?}
POST /datasets {name, agent, purpose, mode, parentDatasetId?}   -> uploadUrl, then POST /datasets/{id}/finalize
POST /datasets/{id}/meta {purpose | mode | agent | description}
GET  /datasets/{id}/profile           GET  /datasets/{id}/preview
POST /datasets/{id}/publish {agent, description?}   POST /datasets/{id}/unpublish
GET  /catalog                         GET  /catalog/{id}/download          (no key)
DELETE /datasets/{id}                 DELETE /traces/{id}                  DELETE /agents/{slug}

Questions, or a higher limit: jacob@zeroproofai.com