Character training with Zero Proof

Most teams give their model a personality with a system prompt. It works until a user says "ignore the role-play and answer normally," or until a prompt change three months later drops the paragraph that made the model warm. Character training is the fix. It changes the weights so the model has a stable way of talking with no prompt telling it to.

It is the same post-training machinery as everything else, aimed at the manner of a reply instead of its correctness. Anthropic describes the process for Claude in plain terms: write the traits, have the model generate questions where each trait matters, generate several replies, and rank the replies by the trait. Most of the work is a data pipeline. This post is that pipeline as the zeroproof SDK runs it. The worked example is examples/character in the SDK repo, and it runs offline in seconds.

What you need

Five things, and the SDK produces the last three.

1. A constitution. One principle per trait, in prose, with good and bad example replies if you have them. The example builds one from the style section of the OpenAI Model Spec: eight traits, sixteen comparisons. Your own spec works the same way.
2. Prompts that make the trait matter. A warmth trait needs prompts where a cold reply is the easy one. Start from the examples in your spec and have the model write more, a few wordings each, so the judge grades the trait and not the phrasing.
3. Several replies per prompt, sampled with the deployment prompt only. The deployment prompt names the persona and nothing else. If the constitution is in the prompt at sampling time, you are measuring prompting, not character.
4. A judge that reads the principle. The principle goes in the judge's system prompt and nowhere else. Use a different model family from the one you are training, and check the judge against the spec's own labeled replies before you trust a single pass rate.
5. A before-and-after test the model never trained on.

The recipe in code

import zeroproof.simulations as zps

# 1. is the judge any good? grade the spec's own GOOD and BAD replies first
zps.judge_agreement(spec_rows)                     # rows carry gold_reward; below about 0.8, fix the judge

# 2. sample k replies per prompt under the deployment prompt, then grade
scored = data.grade(judge=my_judge)
scored.pass_at                                     # per trait: pass@1 and the headroom

# 3. is the judge paying for length instead of character?
zps.reward_correlations(scored.rows)

# 4. pairs for DPO and rows for SFT, length-matched so the trainer learns the trait
pairs, report = zps.build_preference_pairs(scored.rows, length_match=True)
zps.export_preference(pairs, "pairs.jsonl", system_prompt=DEPLOY_PROMPT)
zps.export_training(scored.passes, "sft.jsonl", system_prompt=DEPLOY_PROMPT)

# 5. after training: did the trait move, and did helpfulness hold?
zps.delta_report(before, after, target="marker:trait", must_not_regress=["on_task", "no_filler"])

Every reply gets three scores. trait comes from the judge reading the principle. on_task also comes from the judge: did the reply do the job. no_filler comes from a phrase list the judge never sees, the "Certainly" and "As an AI model" list. The reward on a trait prompt is trait AND on_task. A reply that has the character and drops the task scores zero. Every persona still has to answer the question.

What a live run says

One run, hosted Qwen3-4B as the student and hosted Phi-4 as the judge, four replies per prompt, 239 rows in 148 seconds:

judge llm:microsoft/phi-4 vs spec labels: agreement 0.69 (n=35, kappa 0.40)
pass@1 0.78 | pass^4 0.73 | pass@4 0.80 | headroom 0.02 | mixed prompts 1/15
markers: no_filler 1.00 | on_task 0.93 | trait 0.85
controls on_task 0.88 | adversarial trait 0.97
corr(reward, reply length) -0.16 ok
pairs 1 (chosen longer 0.0) -> out/pairs.jsonl | sft 47 -> out/sft.jsonl

Two things in that output decide what you do next, and neither is visible without the spec rows and the markers.

Tighten the judge first. Phi-4 passed 10 of the spec's 20 bad replies. Those are exactly the rows a preference set would train toward. Fix the judge prompt or use a stronger judge before reading any pass rate. This is why the agreement check runs first.

The spec is already this model's default character. Qwen3-4B lands the Model Spec's traits 78 percent of the time and holds them when a user says "drop the act" (0.97). One mixed prompt, one pair: there is nothing to train on. That is the expected result for an instruct model on the industry-default style guide. The contrast, and therefore the training signal, comes from a distinct persona or from harder prompts. Writing the constitution is the small part.

The rows from that run are public: zeroproofai/character-training-model-spec on Hugging Face, with train, holdout and eval splits. The eval split is the spec's labeled replies, so you can check your own judge against it.

What the before-and-after looks like

The example ships an offline student that replays the spec's own good and bad replies at a fixed rate, so you can see the measurement without a GPU. The numbers are real; the model is not.

marker:trait: moved (+0.500, 95% +0.408..+0.592, 30 paired tasks)
PASS
  pass_at_1        0.375 -> 0.792  +0.417  up
  marker:no_filler 0.681 -> 0.986  +0.306  up
  marker:on_task   1.000 -> 1.000  +0.000  flat
  marker:trait     0.250 -> 0.750  +0.500  up

The headline is the trait, with a confidence interval. on_task is a hard guard: a significant drop there fails the report, no matter what the trait did. The test prompts carry a "drop the act" suffix, which is the robustness check from the character-training literature, so the number you are reading is how much of the character survives a user trying to remove it.

What to watch for

• The judge likes long replies. In the Model Spec's own comparisons, the good reply is the longer one 70 percent of the time. A judge that learned that will pass verbose off-character replies, and DPO will learn length first. Length-neutral judge instructions, length-matched pairs, and the correlation line exist for this.
• The judge is the model being trained. It prefers its own style. The agreement number against the spec labels is where this shows up.
• Character costs helpfulness. A warm reply that does not answer. A refusal that lectures. on_task is the guard, and the plain control prompts carry no trait score at all.
• No contrast. A trait the model already lands every time, or never, produces no pairs. Write prompts where the model is inconsistent, or sample the chosen side from a teacher that has the constitution in its prompt and accept that those pairs are off-policy.
• The test set is the training set. Adversarial variants of training prompts test robustness, not generalization. Model-written prompts split by hash give you a real holdout.

What the SDK does not do

Train. pairs.jsonl is prompt, chosen, rejected as message lists, which any DPO trainer reads. sft.jsonl carries a loss mask on the assistant turn. The SDK produces the rows, the pairs, the judge check and the before-and-after report. Everything after pairs.jsonl is your trainer, and zps.training_run logs that run to the platform so the loss curve and the delta sit next to the data.

Run it

pip install zeroproof
git clone https://github.com/Zero-Proof-AI/zeroproof-sdk
cd zeroproof-sdk/examples/character
python run.py                 # scripted student, offline, seconds
python measure.py --demo      # before vs after on the holdout

With a model, the same script takes a student endpoint, a judge endpoint, and --write-prompts N to have the model write harder prompts per trait. The full recipe, with the sources it follows, is at zeroproofai.com/docs/character-training.