SFT, DPO and PPO on the same model: what controlled comparisons found

Barakaeli Lawuo, Apr 15, 2026

In 2024 a team at the Allen Institute for AI and the University of Washington took one supervised model, Tülu 2 13B (a fine-tuned Llama 2), and trained it further with two preference methods under matched conditions. Same starting weights, same preference data, the same 11 benchmarks.¹ The supervised starting point averaged 56.8 across their categories. Training it with Direct Preference Optimization (DPO) on the UltraFeedback dataset raised that to 61.0. Training it with Proximal Policy Optimization (PPO), using a 13B reward model built from the same UltraFeedback data, gave 62.2. Swapping in a 70B reward model gave 62.8.¹

The gains were uneven. Truthfulness jumped from 56.6 to 69.3 with DPO and 71.5 with PPO. Instruction following went from 44.2 to 52.8 and 54.4. Factuality, measured by MMLU, barely moved: 55.4, 55.3, 56.0.¹ Across five datasets, PPO beat DPO every time, by 0.7 points on average, a gap the authors report as significant at p < 0.05.¹ The cost difference was bigger than the score difference. PPO with a 13B policy and a 13B reward model took 3 days on a TPUv3 pod of size 256, plus roughly 10 hours to train the reward model. DPO on the same data took 9 hours.¹

A second study, from Tsinghua University and collaborators, found a sharper split on competitive programming. On the CodeContests test set, Code Llama 34B after supervised fine-tuning solved 15.2% of problems under the 10@1k protocol (sample 1,000 programs, submit 10). DPO dropped it to 0.0%; the authors report the DPO model \"outputs many meaningless code snippets.\" PPO raised it to 22.4%, above the 16.4% of the 41B AlphaCode system.²

So three training recipes, run on the same model, give different results, and the size of the differences depends on the task. To see why, it helps to write down exactly what each one is minimizing.

What supervised fine-tuning minimizes

A language model is a policy $\pi_\theta(y \mid x)$: a probability distribution over responses $y$ given a prompt $x$, with weights $\theta$. It produces the response one token at a time, so the probability of a whole response is the product of per-token probabilities, each conditioned on the prompt and on the tokens already written.² Supervised fine-tuning (SFT) trains the pretrained model to imitate demonstrations: prompts paired with good responses, which in InstructGPT were written by human labelers.^2,4

Read it term by term. $D_{\text{SFT}}$ is the demonstration set. $y_t$ is the $t$-th token of the demonstrated response, and $y_{<t}$ is everything before it. $\log \pi_\theta(y_t \mid x, y_{<t})$ is how much probability the model gives to the exact token the demonstrator wrote. The minus sign turns that into a loss, so training pushes probability onto the demonstrator's choices, token by token. Nothing in the loss says one response is better than another. It only says: be more like this one.

That gap shows up in practice. In the InstructGPT work, OpenAI's SFT models started to overfit on validation loss after one epoch, yet training for 16 epochs still improved reward model scores and human preference ratings.⁴ Next-token loss on held-out demonstrations and \"people like the output\" are different targets.

The reward model is a pairwise classifier

Preference methods start from a different kind of data: a prompt $x$, two responses, and a label saying which one the annotator preferred. Call the chosen response $y_c$ and the rejected one $y_r$. A reward model $R_\psi(x, y)$ is a transformer whose language-modeling head is replaced by a head that outputs one number per response.¹ It is trained under the Bradley-Terry assumption, which says the probability that $y_c$ beats $y_r$ is the sigmoid of the difference of their scores.²

$D_R$ is the preference dataset. $\sigma$ is the logistic sigmoid, $\sigma(z) = 1/(1+e^{-z})$. The loss only ever sees the difference between the two scores, so adding a constant to every reward changes nothing; the scale is fixed only by the sigmoid. Minimizing it is binary cross-entropy: the model is a classifier for \"which of these two did the human pick,\" and the score it learns is whatever makes that classification work. Anthropic used the same relation to read their scores: a preference model score gap $\Delta$ predicts that the higher-scored response wins with probability $1/(1+e^{-\Delta})$.³

PPO maximizes reward minus a KL leash

Reinforcement learning from human feedback (RLHF) with PPO then trains the policy to produce responses the reward model scores highly, without drifting too far from where it started. The reference policy $\pi_{\text{ref}}$ is usually the SFT model that training begins from.¹

$D_\pi$ is a set of prompts with no responses attached. The responses $y$ are sampled from the policy being trained, which is what on-policy or online means: the model is graded on its own current outputs, not on a fixed file. $R_\psi(x,y)$ is the reward model's score for that response. $D_{\mathrm{KL}}(\pi_\theta \| \pi_{\text{ref}})$ is the Kullback-Leibler divergence, a measure of how far the new distribution has moved from the reference, and $\beta$ sets how hard that movement is penalized.

In implementation the KL term is spread over tokens. Ivison et al. give every token a reward of $-\beta$ times the log ratio of the policy's and the reference's probability for that token, and add the reward model's score only at the last token.¹ A separate value model estimates how much reward a half-written response will end up with, and the gap between what actually came back and that estimate is the advantage $A_t$.¹ PPO then updates the policy with a clipped objective:

$\rho_t$ is 1 when the policy has not changed. If an update would push $\rho_t$ outside $[1-\epsilon, 1+\epsilon]$ in the direction that improves the objective, clipping removes the incentive, so each step stays small.⁶ Ivison et al. used $\epsilon = 0.2$, $\beta = 0.05$ (0.0325 with the 70B reward model), sampling temperature 0.7, and one pass over the prompts; training beyond one epoch \"occasionally resulted in the training collapsing entirely.\"¹ Generating the rollouts took more than 95% of PPO's training time.¹ Anthropic wrote the same trade as a single reward, $r_{\text{PM}} - \lambda_{\text{KL}} D_{\mathrm{KL}}$, with $\lambda_{\text{KL}} = 0.001$, a value they say \"might actually be wholly unnecessary.\"³

DPO skips both the reward model and the sampling. Rafailov et al. showed that the optimum of the KL-regularized objective implies a reward equal to $\beta \log$ of the policy-to-reference probability ratio, plus a term that depends only on the prompt.⁵ Substitute that into the Bradley-Terry loss and the prompt term cancels, leaving a loss on the policy alone: $-\log\sigma$ of $\beta$ times the chosen response's log ratio minus the rejected one's.^1,2,5 It trains on a fixed file of pairs, which is what offline means here.

Reference policy

The frozen model the KL penalty measures distance from, usually the SFT checkpoint.

On-policy (online)

Training data is generated by the current model during training, as in PPO.

Offline

Training uses a fixed, pre-collected set of responses, as in standard DPO.

Distribution shift

The responses in the preference data look unlike what the model itself would write.

Data moved the scores more than the algorithm did

Before comparing algorithms, Ivison et al. ran DPO on 14 preference datasets. The overall average ranged from 51.5 (Chatbot Arena 2024) to 61.0 (UltraFeedback with per-aspect ratings).¹ That spread is about 9.5 points; the average PPO-over-DPO gap on matched data was 0.7. Their conclusion ranks the factors in that order: preference data quality first, then algorithm, then reward model, then the choice of training prompts.¹

Some details inside that table are worth knowing. Datasets built from per-aspect scores (a rater scores helpfulness, harmlessness and so on separately, and the scores are averaged) did best. HelpSteer, with 9,270 pairs, scored 58.0; HH-RLHF, with 158,530, scored 58.1.¹ The choice of which response counts as chosen mattered more than how good the responses themselves were.¹ And DPO on Chatbot Arena votes cut the safety score from 91.8 to 67.3 (2023 data) and 58.1 (2024 data), which the authors read as volunteers there tending to prefer more toxic completions.¹ Preference training learns whatever the labels reward, including things nobody meant to ask for.

Xu et al. found DPO especially sensitive to where its data came from. On the SafeRLHF dataset, a Llama 2 7B model fine-tuned on Alpaca and then trained with DPO reached a 55.4% safety rate. First fine-tuning the same model on SafeRLHF's own safe responses, so the preference data looked more like its outputs, raised DPO to 71.8% and its helpfulness score from -4.19 to -1.62. PPO reached 99.5% with a helpfulness of +1.69.² Their theory result explains the mechanism: every policy PPO can reach also minimizes the DPO loss, but not the reverse. In a three-response toy case where the data compares only the first two, DPO can put 90% of its probability on the third response, which appears nowhere in the data. The KL term rules that out for PPO because the reference gives it zero probability.²

PPO also needed tuning to win. On APPS introductory problems, a baseline PPO run with batch size 64 dropped Code Llama 34B from 38.6% to 18.0% pass@5. Adding advantage normalization brought it to 38.1%, large batches to 42.3%, and slowly updating the reference model with an exponential moving average to 44.4%.² A comparison of \"DPO versus PPO\" is also a comparison of how carefully each was configured.

A bigger reward model mostly helped math

If the reward model is PPO's teacher, a better teacher should help. Ivison et al. tested this with reward models at 13B and 70B, trained on UltraFeedback alone or on a larger mix of six datasets. On RewardBench, a direct test of whether a reward model picks the human-chosen response, the 13B mixture model scored 79.8 against 61.0 for the 13B UltraFeedback model. Used inside PPO, it produced a slightly worse policy: 61.6 average against 62.2.¹ Only the 70B UltraFeedback reward model improved the overall average, to 62.8, and most of that came from GSM8K math, 53.0 to 58.0.¹ With prompts drawn from the GSM8K training set and a 70B reward model, GSM8K accuracy rose from the SFT model's 46% to 62%.¹ Better reward-model benchmark scores did not reliably turn into a better policy, a result the authors describe as surprising.

Anthropic's two datasets pulled in opposite directions

Bai et al. at Anthropic, in 2022, collected two separate kinds of human comparisons, mostly on 52B models. In the helpfulness task, crowdworkers asked for help and picked the more helpful and honest of two replies. In the red-teaming task, they tried to provoke harmful output and picked the more harmful reply.³ The base set held 44k helpfulness and 42k red-team comparisons.³ They trained preference models at seven sizes from 13M to 52B parameters and found accuracy rising roughly log-linearly with both model size and data size.³

Then they mixed the two datasets in 10% steps from all-helpful to all-harmless, holding the total at 42k comparisons. A preference model trained only on one kind of data scored well below chance on the other.³

The paper also reports that larger preference models are less sensitive to the exact mixture, possibly because they are better at telling reasonable requests from harmful ones.³ The tension reached the policies too. Early in the project, many RLHF policies gave the same exaggerated answer to anything remotely sensitive, such as recommending therapy whenever a user expressed any displeasure. The authors attribute it to over-optimizing harmlessness and under-optimizing helpfulness: to score well on red-team prompts, a reply like \"I can't answer that\" is probably enough, and that is easier to learn than being helpful.³

They trace part of the cause to their own labels. Because red-teamers picked the worse reply, the harmlessness data only told the model what not to do, and never showed what a good answer to a harmful request looks like beyond the first turn.³ They partly fixed the problem by giving RLHF a larger share of helpfulness prompts, and proposed collecting data where workers pick the best response instead.³ On capability, they report a split by size: small models paid an \"alignment tax\" on NLP evaluations after RLHF, while their 13B and 52B models did better zero-shot and the same few-shot.³

What the Tülu study did not measure

Ivison et al. list their own limits. The results come from one model suite, Tülu 2, on two base models, Llama 2 and Llama 3, and do not cover multilingual use. They also note that while PPO does well, it costs much more compute than DPO, and that they \"don't explicitly measure the relative computational cost\" of the methods, for example in FLOPs.¹ The 0.7-point average win for PPO on their benchmarks therefore has no cost column beside it, apart from the wall-clock figures in their appendix: 3 days of PPO on a size-256 TPUv3 pod against 9 hours of DPO for the same 13B model.¹