Reference · vocabulary
From the book to the platform
Nathan Lambert's RLHF book, chapter by chapter, against what is here. The book's word, the platform's word, the call. Where the platform does not have the thing, the row says so.
Chapter by chapter
| Chapter | The book says | Here | Call or page | On this site |
|---|---|---|---|---|
| 01 Introduction | RLHF | measure, simulate, train, prove | - | Workflow |
| 02 A Tiny History of RLHF | history of RLHF | not here | - | Post-training |
| 03 Training Overview | SFT, reward model, RL stages | Train button: SFT, GRPO, DPO | zps.train(dataset, method="sft" | "grpo" | "dpo") | Training |
| 04 Instruction Fine-Tuning | instruction tuning, loss masking | SFT rows with a loss mask | zps.export_training(rows, mask_mode="assistant" | "final") | Training rows |
| 05 Reward Modeling | reward model | trained on the pairs, then a judge | zps.train(ds, method="rm"); zps.reward_model(run) | Reward model |
| 06 Reinforcement Learning | policy gradient: PPO, GRPO | GRPO only, no PPO | zps.train(dataset, method="grpo") | RL, end to end |
| 07 Reasoning and Inference-Time Scaling | RLVR, pass@k | verifiable reward, pass@k with intervals | zps.pass_at(rows, k=8) | Three numbers |
| 08 Direct-Alignment Algorithms | DPO | DPO on pairs | zps.train(dataset, method="dpo") | DPO |
| 09 Rejection Sampling | rejection sampling | best completion per prompt by reward, with chance controls | zps.optimize(rows, mode="sft", select="top_per_prompt", min_reward=0.7) | Prune, then publish |
| 10 The Nature of Preferences | human preferences, annotators | judge checked against hand labels | zps.judge_agreement(rows, gold="gold_reward") | Trust the numbers |
| 11 Preference Data | preference data, margin, length bias | pairs with margin and length gap | zps.build_preference_pairs(rows); zps.export_preference(pairs) | Training rows |
| 12 Synthetic Data & Distillation | synthetic data, on-policy distillation | simulations; distillation not here | zps.simulate(agent, spec=...) | Agent simulations |
| 13 Tool Use and Function Calling | tool use, function calling | tool-call efficiency marker | zps.conduct_grade(trajectory) | Tool-call efficiency |
| 14 Over-Optimization | over-optimization, reward hacking | reward-hack scan plus style drift markers | zps.style_report(rows); zps.refusal_report(benign_rows) | Trust the numbers |
| 15 Regularization | KL penalty, reference model | KL measured, not applied | zps.mean_kl(rows); zps.logprob_report(rows) | RL runs |
| 16 Evaluation | evals, contamination | holdout by task, decontamination, before and after | zps.decontaminate(rows, against); zps.delta_report(before, after) | Trust the numbers |
| 17 Model Character & Products | character training, model spec | character training | examples/character | Character training |
Calls are import zeroproof.simulations as zps, SDK 0.30. examples/ paths are in the SDK repo. Definitions on the glossary.
Not here
not here
GRPO only, no PPO
simulations; distillation not here
KL measured, not applied
The LLM judge is the default reward; a reward model is one Train press away. No PPO: GRPO and DPO. No on-policy distillation loop. KL is a number on the run page, not a term in the loss; the trainer applies it. Over-optimization is a scan for reward-length and reward-tool-count correlation, not a monitor during training.
Questions, or a higher limit: jacob@zeroproofai.com