Inside the Model: How Models Learn

Next-token prediction and denoising learn a distribution by restoring something the training process removed. RLHF and DPO then tilt that distribution toward the answer a person preferred.

TLDR

Pretraining needs no hand-written class labels. The held-out piece of a real example is the label.
Next-token prediction trains a language model. Each position scores the whole vocabulary, and the loss is its surprise at the true continuation.
Denoising trains a diffusion model. Corrupt a latent by a random amount, and ask the network to undo that corruption. Sampling walks the same task from full noise toward a clean image.
Those objectives learn what the collection tends to do. Instruction following and taste are a second stage.
RLHF fits a reward model to human comparisons, then optimizes the language model against that score, with a leash back to the starting policy.
DPO trains on the same pairs directly. The preferred answer rises relative to the rejected one, measured against a frozen reference.

The label is the data

A classifier in the textbook sense needs someone to say which class a row belongs to. Text and images at web scale arrive without that column. They arrive as documents and as pictures.

The training trick is to damage a complete example and ask the model to repair the damage. The original example is the label. The loss is the error in the repair.

Two repairs cover the models in this series. A language model repairs a missing next token, using the contextual vector from part 1. A diffusion model repairs a noised latent. The sampler in part 2 is that repair, run from the noisy end.

Next-token prediction

Radford et al. (2019) train GPT-2 by scoring the next token. Take a sequence and hide the future. At position t the model may see tokens 1 through t. The causal mask is what enforces that. The model must then put a probability on every entry in the vocabulary, as its guess of token t+1.

The scores come from a linear layer on the contextual vector, one number per vocabulary entry. A softmax turns the scores into probabilities. The loss on that step is the negative log of the probability on the true next token. A confident, correct guess costs almost nothing. A confident wrong guess costs a great deal, because the log of a tiny probability is a large negative number and the loss flips the sign. Averaged over a corpus, this is cross-entropy, the language-modeling objective in that paper.

During training the model is shown the true prefix, including when its own previous guess would have been wrong. Williams and Zipser (1989) name that setup teacher forcing. The model learns a one-step repair against real context. At sampling time it has to continue from prefixes it wrote itself. A small early miss can steer a long answer, because each later repair conditions on the miss.

Whatever helps prediction can be absorbed: spelling, syntax, a name that tends to follow another name, code that tends to compile, the shape of an argument in the essays the crawl contained. The objective asks for a plausible continuation. A true sentence and a fluent invention can both score well when both show up in text of that kind. The preference stage is how a product picks between them.

Denoising

Ho, Jain, and Abbeel (2020) train a diffusion model on a different kind of damage. Take a clean sample, draw a noise level t, and mix it with Gaussian noise according to a fixed schedule. A small t leaves the sample almost clean. A large t leaves almost pure noise. The network sees the noisy sample and the level t, and it predicts the noise that was mixed in. The loss is the squared error of that prediction. In the latent models of part 2, the sample is the autoencoder's latent and the target is the same noise prediction.

Some models predict the clean latent. Some predict a blend of the clean latent and the noise. The target is the corruption in every case. Stable Diffusion's usual target is the noise itself.

Because t is chosen at random for every example, one network learns every noise level. Sampling uses that skill in order. Start from pure noise, predict the corruption, step toward the clean image, and repeat. The walk can take fewer steps than the training schedule defined. The sampler is allowed longer strides. The text condition is present in both phases. In training, the network learns to denoise toward images that match a caption. In sampling, the caption steers which clean image the walk approaches.

The family resemblance is the lesson. Next-token prediction fills a discrete hole at the end of a prefix. Denoising fills a continuous hole spread through the latent. Both turn an unlabeled collection into a training signal by holding part of a real example back.

What pretraining leaves undone

A model that has only minimized those losses continues in the voice of its data. Asked a question, it may answer. It may write another question. It may extend a forum thread. All three are real continuations of text it has seen. The loss has no term for "do what this person asked," "be brief," or "refuse this."

Those are preferences about behavior. They live in comparisons between answers. The raw corpus does not mark them. The second stage starts from the pretrained model and uses a much smaller set of human judgments.

RLHF

Christiano et al. (2017) train a reward from human comparisons, then optimize a policy against it. Ouyang et al. (2022) bring that recipe to instruction-following language models in three moves.

1
Supervised fine-tuning Writers demonstrate the behavior: a prompt, and the answer the product should give. The same next-token loss, on those demonstrations only.
2
A reward model People compare two answers to the same prompt. A second model learns to score the preferred one higher.
3
Policy optimization The language model is updated to raise that score, and a penalty keeps it close to the supervised model.

The supervised model is already a policy: a distribution over replies that answers, where the pretrained model might have continued a web page. The reward model compresses the comparisons. Its training rule is pairwise. The probability that one answer beats the other should track how often people said it did. A logistic curve on the gap between the two scores can express that. The number it outputs is an approximation of a preference, with all the roughness of a model trained on a finite set of votes.

The policy is then optimized to increase the reward. Proximal policy optimization is the usual algorithm. It takes steps that raise the score and that stay inside a trust region of the current policy. A KL penalty pulls the result back toward the supervised model. Improvement has to be a kind of reply the supervised model could already have given some probability to.

Without that leash, the policy finds phrasing the reward model overrates. The reward model is a proxy for a person. A proxy can be gamed. The name for that failure is reward hacking, and the KL term is the standard defense against it.

DPO

Rafailov, Sharma, Mitchell, Ermon, Manning, and Finn (2023) use the same pairs and skip the separate reward model and the reinforcement-learning loop. Their derivation starts from the best policy for a reward minus a KL penalty. That policy has a closed form. In the closed form, the reward can be rewritten as the gap in log probability between the trained policy and a frozen reference, usually the supervised model. Substitute that expression into the pairwise rule, and the reward model drops out. A classification loss on each pair is what remains.

The loss falls when the policy raises the probability of the preferred answer and lowers the probability of the rejected one, judged relative to the reference. A coefficient β sets how far that gap may grow. A large β stays near the reference. A small β follows the preferences further.

loss = −log σ( β [log π(y_w|x) − log π_ref(y_w|x)] − β [log π(y_l|x) − log π_ref(y_l|x)] )

y_w is the answer a person preferred. y_l is the one they set aside. σ is the logistic curve. The reference model in the denominator is the leash, written directly into the loss.

The pairs still have to be the comparisons you mean. A rejected answer should be one you want less of: an instruction ignored, a reply that wanders, a claim in the tone of certainty with nothing behind it. If the rejected column is full of answers that were acceptable, the loss spends its gradient suppressing them. DPO has fewer moving parts than the RLHF loop. It depends just as hard on what people were asked to compare.

What to take with you

Unstructured data is enough when the question is "what usually comes next," or "what did this noise cover." The model learns the shape of the collection it was given. Human preference is a second dataset, much smaller, whose label is a comparison.

RLHF compresses the comparisons into a score and optimizes against the score, leashed to the supervised model. DPO writes the leash into the loss and trains on the pairs. Both raise the preferred replies, including the facts written inside those replies, at the scale of a fine-tune. A body of knowledge that appears in neither the pretraining collection nor the preferred answers is still absent. The second stage tilts a distribution. The first stage is where that distribution came from.

Sources

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners. OpenAI, 2019.
Ronald J. Williams and David Zipser. A Learning Algorithm for Continually Running Fully Recurrent Neural Networks. Neural Computation, 1989. Teacher forcing.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. arXiv:2006.11239, 2020.
Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep Reinforcement Learning from Human Preferences. arXiv:1706.03741, 2017.
Long Ouyang and colleagues. Training language models to follow instructions with human feedback. arXiv:2203.02155, 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arXiv:2305.18290, 2023.