Reward ModelsOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Reward Models

Cross-entropy and SFT both need a single correct answer per input. Alignment is messier than that: "helpful", "honest", and "concise" are tastes, not labels. A reward model (RM) bridges the gap — it learns a scalar r(prompt, response) that mimics how a human would rank competing answers. Once you have a reward, you can optimise against it with RL (next lesson), or use it directly as a re-ranker over candidate generations. The reward model is the objective slot for alignment: it's how you define "good" when there's no single correct label to regress on.

Real-world examples: InstructGPT/ChatGPT trained an RM from ~30 k labeller-ranked comparisons before any RL kicked in; Anthropic's RLHF and Constitutional AI loops train RMs on chosen-vs-rejected pairs; LLaMA-2-Chat learned two RMs (helpfulness + safety) and combined them at PPO time; open-source projects (TRL, OpenRLHF, Argilla) all centre on a reward_model.fit(pairs) step.

The preference dataset

You can't ask labellers for an absolute reward — humans are terrible at numbers and great at comparisons. The unit of data is therefore a pairwise preference:

D={ (xi,  yc,i,  yr,i) }i=1N,\mathcal{D} = \{\,(x_i,\; y_{c,i},\; y_{r,i})\,\}_{i=1}^{N},

where for the same prompt xx the labeller marked ycy_c (chosen) as better than yry_r (rejected). Where do these pairs come from?

  • Two candidate completions from the SFT model with different sampling seeds, ranked by a human.
  • Larger N-way rankings (4–7 responses) decomposed into all (N2)\binom{N}{2} ordered pairs.
  • Synthetic preferences from a stronger model judging a weaker one (RLAIF / AI feedback).
  • Off-policy mixes: SFT model vs the previous RLHF checkpoint, vs human-written gold.

A typical RM dataset is 20 k – 200 k pairs — much smaller than pretraining or SFT data, because each pair is expensive to collect and the signal is dense.

The Bradley–Terry objective

Bradley–Terry (1952) is the workhorse pairwise-ranking model. It assumes every item has a latent score and that the probability one beats another is a logistic function of the score difference:

P(yc≻yr∣x)  =  σ ⁣(rθ(x,yc)−rθ(x,yr)).P(y_c \succ y_r \mid x) \;=\; \sigma\!\big(r_\theta(x, y_c) - r_\theta(x, y_r)\big).

Maximising the log-likelihood of the observed preferences gives the reward-model loss:

LRM(θ)  =  − E(x,yc,yr)∼D[log⁡σ ⁣(rθ(x,yc)−rθ(x,yr))].\mathcal{L}_{\text{RM}}(\theta) \;=\; -\,\mathbb{E}_{(x,y_c,y_r)\sim\mathcal{D}}\Big[\log \sigma\!\big(r_\theta(x,y_c) - r_\theta(x,y_r)\big)\Big].

A few properties worth internalising:

  • The loss only depends on the margin Δr=rθ(x,yc)−rθ(x,yr)\Delta r = r_\theta(x, y_c) - r_\theta(x, y_r). Absolute reward values are unobservable; the model is identifiable only up to a per-prompt constant.
  • Δr=0\Delta r = 0 gives σ(0)=0.5\sigma(0) = 0.5 and loss log⁡2≈0.693\log 2 \approx 0.693. That's your "random" baseline.
  • Doubling all rewards halves the temperature of the implicit comparison — RMs are scale-sensitive, which matters later when you plug them into PPO.

Reward model: pairwise preference → reward margin

prompt: “Explain entropy to a five-year-old.”-3-2-10123rmargin = 1.60r_r = -0.40Rejected ✗ long, jargon-heavy replyr_c = 1.20Chosen ✓ short, vivid analogyσ(margin) — chosen-wins probability00.51-6-3036z
margin r_c − r_r1.60
P(chosen wins)83.2%
BT loss −log σ(Δr)0.184

Two candidate responses to one prompt, the chosen one (✓) and the rejected one (✗). Each gets a scalar reward from the reward model. The Bradley–Terry objective trains the model so the chosen reward sits above the rejected — the wider the margin, the higher the win probability σ(r_c − r_r) and the lower the loss −log σ(r_c − r_r).

Why the sigmoid, not the difference?

You could imagine training on the raw margin rc−rrr_c - r_r — bigger is better. But that loss is unbounded: the optimiser can drive the margin to infinity and never stops. The sigmoid squashes the signal into [0,1][0, 1], so confident-correct pairs contribute almost no gradient and the model focuses on the hard ones. The same trick that powers logistic regression.

A scalar reward head on a frozen trunk

The reward model is almost a language model. The recipe:

  1. Start from the SFT checkpoint (the model just learned the task distribution, so its features are already useful).
  2. Replace the final language-modelling head with a tiny linear head that produces a single scalar per token.
  3. Feed in the prompt + response, take the last-token hidden state, and project it to one number — that's rθ(x,y)r_\theta(x, y).
  4. Optionally freeze the trunk and only train the head (fast, sometimes works); more commonly, fine-tune the top few layers + the head for a few epochs.
rθ(x,y)  =  w⊤hlast(x,y)+b,w∈Rd,  b∈R.r_\theta(x, y) \;=\; w^\top h_{\text{last}}(x, y) + b, \qquad w \in \mathbb{R}^{d}, \; b \in \mathbb{R}.

For chat models, the prompt and response are concatenated through the chat template (same template the policy will use at RL time), and the scalar is taken at the final <|assistant|> end-of-turn token.

Why start from SFT? A from-scratch reward head has no idea what "good" looks like; an SFT-initialised trunk already encodes "fluent, on-topic, follows instructions" in its representations. The RM only has to learn the delta between mediocre and great. This is why RMs need orders of magnitude less data than the base pretraining.

Reward hacking and Goodhart's law

The RM is a proxy for human preference, never the real thing. As soon as you optimise against it (PPO, best-of-N, re-ranking), the policy starts finding gaps between the proxy and the true preference. This is reward hacking: high reward, low actual quality.

Classic failure modes from the literature:

  • Length bias. Human raters prefer longer answers on average → the RM rewards length → PPO produces verbose, padded responses for everything. LLaMA-2-Chat had to length-normalise reward; many recipes also add a length penalty.
  • Sycophancy. If labellers preferred answers that agreed with their stated opinion, the RM learns to flatter. Now your policy is dishonest.
  • Formatting tells. Bullet points, markdown headers, "Sure! Here is…" preambles — the RM may latch on to these as cheap features that correlate with chosen.
  • Refusal calibration. Over-cautious RMs reward refusals → the policy refuses everything mildly edgy → user trust collapses.

The Goodhart-style statement: once a measure becomes a target, it ceases to be a good measure. The reward model is a snapshot of preference under a specific data distribution; pushing the policy hard against it drifts both off-distribution and the proxy gap widens.

Mitigations in practice

Standard countermeasures: (1) KL penalty during PPO to keep the policy close to the SFT model (next lesson); (2) reward ensembling — train several RMs with different seeds and use the disagreement as an uncertainty signal; (3) regular RM refreshes on freshly collected preferences; (4) human eval as the final word — never ship on RM score alone.

Evaluating a reward model

Before plugging an RM into PPO, sanity-check it.

  • Pairwise accuracy on held-out pairs. Fraction of the time rθ(x,yc)>rθ(x,yr)r_\theta(x, y_c) > r_\theta(x, y_r) on data the model hasn't seen. 65–75 % is typical for current open models; near 100 % usually means a leak.
  • Calibration. Bin pairs by predicted σ(Δr)\sigma(\Delta r) and check that the empirical win rate of the chosen response matches.
  • Slice metrics. Accuracy on different subsets (by topic, length bucket, refusal vs answer) exposes systematic biases like the length tell.

Exercise

The Bradley–Terry loss for a preference pair is $\mathcal{L} = -\log \sigma(r_\theta(x, y_c) - r_\theta(x, y_r))$. What does this loss depend on?

Need a hint?

What changes if you add the same constant to both r_c and r_r?

Exercise

The reward model assigns r_c = 1.5 to the chosen response and r_r = −0.5 to the rejected one. What is the implied probability (in percent) that the chosen response wins, σ(r_c − r_r)?

0%50.00%100%
Need a hint?

Compute the margin Δr = r_c − r_r = 2, then σ(2) = 1 / (1 + e^{-2}).

Exercise

After PPO against a reward model, your policy generates needlessly long, padded answers — but the reward model score keeps going up. What's the most likely diagnosis?

Need a hint?

What's a reliable surface feature that labellers might have rewarded?

Common mistakes

  1. Training the RM on a single response per prompt. You need pairs (or rankings); a regression target on absolute ratings is far noisier.
  2. Mismatching the chat template between RM training and PPO. The RM sees one prompt format and the policy emits another → garbage reward.
  3. Ignoring length normalisation. Without it, the RM happily rewards filler, and PPO obediently emits filler.
  4. Pushing PPO until RM score saturates. That's the textbook hacking regime — the policy is over-fit to the proxy. Use the KL budget and human eval as stopping criteria.
  5. Re-using the SFT validation set as the RM eval. Different task, different distribution — collect fresh preferences for the RM eval split.

Key takeaways

  • A reward model maps (prompt, response) → scalar by learning from pairwise preferences, not absolute ratings.
  • The Bradley–Terry loss −log⁡σ(rc−rr)-\log \sigma(r_c - r_r) trains the model to push the margin between chosen and rejected.
  • Architecture: SFT-initialised trunk + a one-dimensional linear head on the last-token hidden state. Absolute rewards are arbitrary; only differences matter.
  • Reward hacking is the inevitable failure mode when you optimise against a proxy. Counter it with KL penalties, ensembles, fresh data, and human eval — the RM is a tool, not the truth.

Related concepts

  • Supervised Fine-Tuning: the checkpoint the RM is initialised from, and the loss it stops using.
  • PEFT: LoRA & QLoRA: RMs are usually trained with the same parameter-efficient tricks: tiny adapters on a frozen base.
  • Markov Decision Processes: the reward function the next lesson plugs into PPO is exactly rθr_\theta from this lesson.

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelated
Reward ModelsAttention & TransformersReinforcement LearningFrom Policy Gradient to R…LLM EvaluationML Systems Design & Probl…Supervised Fine-Tuning & …PEFT: LoRA & QLoRAMarkov Decision Processes