# Zero Proof Labs · Trace Ingest

Send your agent's OpenTelemetry traces to Zero Proof. They land on your
account in the same JSONL rollout shape the Simulations SDK produces, so the
analyzer can score them: prompt grouping, gradient support, pass rate,
reward-hack artifacts, and a training-regime verdict. The data is yours, to
pull back out and train your own models on.

## Endpoint

```
POST https://wch04mgo2k.execute-api.us-east-1.amazonaws.com/v1/traces
X-Api-Key: zp_...            # any key from your platform page
Content-Type: application/json
Body: OTLP/HTTP JSON (ExportTraceServiceRequest)
```

Response `202`:

```json
{ "datasetId": "ds_0123…", "dataset": "prod-refunds", "day": "2026-08-24",
  "rows": 412, "bytes": 918_004 }
```

Fire-and-forget: one call, one 202. No presigned URL, no finalize step,
nothing to poll.

**Batches append.** Everything sent under one `zeroproof.dataset` name on one
UTC day is one dataset, so the default five-second exporter flush does not
turn into thousands of them. Each batch is stored as its own part file named
after a hash of its contents, which means an exporter retry overwrites rather
than duplicating: a duplicated rollout would quietly skew the support and
pass-rate numbers you are collecting this data for.

## One-time setup (any language)

Three environment variables. No Zero Proof code in your app.

```
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://wch04mgo2k.execute-api.us-east-1.amazonaws.com/v1/traces
OTEL_EXPORTER_OTLP_HEADERS=x-api-key=zp_...
OTEL_EXPORTER_OTLP_PROTOCOL=http/json
```

Notes:

- Use the signal-specific `..._TRACES_ENDPOINT`, which is sent as written.
  The generic `OTEL_EXPORTER_OTLP_ENDPOINT` appends `/v1/traces` for you, so
  with that one, drop the path.
- `http/json`, not the default `http/protobuf`. The endpoint parses the OTLP
  JSON wire format; protobuf batches get a `415` with this instruction.
- Keep batches under 8 MB (a typical SDK flush is far below that). Bigger
  batches get `413`.
- gzip is accepted, not required.

## Tagging your spans

Every span is stored, and a trace is assembled across batches, so a turn whose
spans arrive in three flushes is one run rather than three. A rollout row is
derived from each trace for the analyzer to read.

Where two spellings are listed, prefer the neutral one: an agent that emits the
OpenTelemetry testing namespace carries no vendor attribute names in its
source, which matters if it ships to other people.

| Attribute | Where | Meaning |
| --- | --- | --- |
| `zeroproof.dataset` | resource | Dataset name (default `traces`) |
| `test.case.name` (or `zeroproof.scenario_id`, `scenario.id`, `dataset.item.id`) | any span | Groups repeats of the same task. Without it every run is a group of one |
| `zeroproof.score` (or `zeroproof.reward`) | any span | A score your harness or judge produced, in any units |
| `zeroproof.score.pass_at` | any span | The bar, in the same units as the score. Without it the score is recorded but not judged |
| `zeroproof.score.max` | any span | Optional. Only affects how the score is shown |
| `zeroproof.scores.<name>` | any span | A named measurement, so several judges can report on one run |
| `gen_ai.agent.name` | agent span | Names the agent (informational) |
| `gen_ai.request.model` | LLM span | Model used (recorded per row) |
| `gen_ai.usage.input_tokens` / `gen_ai.usage.output_tokens` | LLM span | Token counts (summed per row) |
| `gen_ai.tool.name` + `gen_ai.tool.call.arguments` / `gen_ai.tool.call.result` | tool span | Tool call trace (recorded per row) |

Prompts and responses are read from `gen_ai.input.messages` /
`gen_ai.output.messages` (JSON string or array) when present, falling back to
`gen_ai.prompt` / `gen_ai.completion`. Instrumentation libraries that follow
the GenAI semantic conventions emit these already; the `zeroproof.*` ones you
set yourself.

## Python helper

```python
# live exporter: same three variables, set from code
import os
from zeroproof.ingest import otel_env
os.environ.update(otel_env("zp_...", dataset="prod-refunds"))

# or replay a batch you already have on disk
from zeroproof.ingest import ingest_traces
ingest_traces("zp_...", "traces.json", dataset="prod-refunds")

# what has landed so far
from zeroproof.ingest import list_traces
for t in list_traces("zp_...")["traces"]:
    print(t["name"], t["rows"], t["sizeBytes"])
```

## Evals

An eval run is these same traces, tagged with the set and run they belong to.
There is no separate store and nothing to upload: your agent runs its cases
wherever it already runs, and the scoring happens on our side so that fixing a
check re-scores every run you have already done.

| Attribute | Where | Meaning |
| --- | --- | --- |
| `test.suite.name` (or `zeroproof.eval.set`) | any span | Makes the run an eval and names the set |
| `test.run.id` (or `zeroproof.eval.run`) | any span | One id per run, so two runs can be compared case by case |
| `test.case.checks` (or `zeroproof.eval.checks`) | root span | JSON array of checks to score this case against |
| `test.case.evidence.*` (or `zeroproof.evidence.*`) | any span | Facts only your machine can see, such as whether tests passed |

Checks are mechanical and evaluated from the stored trace:
`no_error`, `tool_called`, `tool_not_called`, `max_duration_ms`,
`max_tool_calls`, `output_contains`, `output_matches`, `evidence_equals`,
`evidence_true`, `evidence_contains`, `score_at_least`. A case is green only
when every check on it passes.

If your agent already names these things its own way, declare the mapping on
the resource instead of changing code: `zeroproof.map.case`, `.set`, `.run`
name the attribute to read. Resource attributes come from
`OTEL_RESOURCE_ATTRIBUTES`.

## Scoring after the fact

A judge answers after the turn it is judging has closed. Send what it decided
against the trace id, any time later:

```
POST /v1/scores
X-Api-Key: zp_...

{ "traceId": "...",
  "scores": [
    { "name": "score",       "value": 0.4, "pass_at": 0.7, "source": "my-judge" },
    { "name": "helpfulness", "value": 3, "max": 5 },
    { "name": "tone",        "label": "curt" }
  ] }
```

The shortest form is `{"traceId": "...", "name": "score", "value": 0.9}`, and a
batch across many traces works by putting `traceId` on each item.

Measurements are named, so several judges and a human label sit side by side
rather than overwriting one another, and the names are yours: nothing has to be
declared first. Units are yours too, since `pass_at` is read in the same units
as `value`, so three out of five and 85 out of 100 both work with nothing
rescaled. A label with no number counts as a measurement.

Sending the same name again is a correction rather than a duplicate, so a judge
you have fixed can say so. A batch containing an unknown trace id returns `207`
and names it, rather than discarding the measurements that were fine.

## Getting it back

Everything below takes the same `X-Api-Key`, and only ever sees your own
account's data.

```
GET    /traces                  runs, newest first, filterable
GET    /traces?since=<ms>       only what changed since, for following a stream
GET    /traces/{traceId}        one run and every span in it
DELETE /traces/{traceId}        forget one run
GET    /evals                   eval sets on the account
GET    /evals/{set}             runs in a set, with the case-by-case grid
GET    /datasets/{id}           one dataset's metadata
GET    /datasets/{id}/download  presigned URLs: `parts`, concatenate for the JSONL
GET    /datasets/{id}/report    the analyzer's signal report
DELETE /datasets/{id}           removes the registry row and every stored part
```

`GET /traces` narrows by `from`, `to`, `status`, `service`, `model`, `tool`,
`scenario`, `session`, `hasReward`, `minDuration` and free text `q`. The
response says whether the scan reached the end of the window, so an empty page
can be told apart from a scan that stopped early.

Each trace also becomes one JSONL row for training: `scenario_id`, `prompt`,
`final_text`, `reward`, `tool_trace`, `info.model`, `info.*_tokens`. One row
per trace however many batches it arrived in.

To score a dataset, send it to the analyzer:
`POST https://zeroproofai--zeroproof-analyzer-api.modal.run/analyze` with
`{"datasetId": "ds_..."}` and `X-Api-Key`.

Without a score the dataset is `PROMPTS ONLY`: you still get shape and prompt
metrics, but no RL-health numbers. Add one, from your own harness or a judge,
to unlock them.

## Limits

| Limit | Value | Over it |
| --- | --- | --- |
| One batch | 8 MB | `413`, the exporter's own batch size stays well under |
| One dataset, one day | 1 GB | `429` until 00:00 UTC, or use another dataset name |
| Account storage | 5 GB, shared with Simulations datasets | `507`, delete a dataset |

The caps exist because an exporter is a continuous writer and a leaked key
would otherwise be an unbounded one. The daily token quota applies to model
calls, not to ingest.

## What we do with it

We store it and analyze it for you. Traces stay until you delete them, we do
not train on them, and one account's data is never readable with another
account's key. Message content is stored in full, because the gaps you are
looking for live in the text; if that is not acceptable for a given service,
leave `gen_ai.input.messages` and `gen_ai.output.messages` off its spans and
you still get structure, rewards, tool calls, and token counts.