Core LLM RL Algorithms: PPO, DPO, GRPO

This note focuses on three central algorithms: PPO as the classic online RLHF workhorse, DPO as a common offline preference optimization method, and GRPO as a critic-free RL method often used in reasoning-model training. Many variants deserve their own post; this page focuses on the core ideas.

0. Memorize This Table First

Mnemonic:

PPO  = online sampling + reward model/function + critic baseline + KL/clip
DPO  = offline preference pairs + reference model + classification-style loss
GRPO = online multi-sampling per prompt + group-relative reward + no critic

1. Shared Foundation: Policy Gradient

An LLM can be viewed as a policy:

pi_theta(y | x)

Given a prompt x, the model generates an answer y. The RL objective is to maximize expected reward:

maximize_theta E_{x ~ D, y ~ pi_theta(. | x)}[R(x, y)]

The simplest update direction is:

grad J(theta) ~= A(x, y) * grad log pi_theta(y | x)

Intuition:

• If an answer is better than the baseline, increase its generation probability.
• If an answer is worse than the baseline, decrease its generation probability.
• Real training must reduce variance, control KL drift, and handle long sequences, which is why more stable objectives such as PPO and GRPO are used.

2. PPO: Classic RLHF

The standard PPO-based RLHF pipeline is:

SFT model
-> train reward model from preference data
-> online rollout with current policy
-> PPO update policy under KL constraint

A common LLM PPO reward is:

reward = reward_model_score - beta * KL(pi_theta || pi_ref)

PPO training usually has four roles:

The core PPO objective can be understood as:

ratio = pi_theta(token) / pi_old(token)
objective = min(ratio * advantage, clip(ratio, 1-eps, 1+eps) * advantage)

Here pi_old is the old policy that produced the rollout samples. During the PPO update, the importance ratio compares the current policy with the policy used at sampling time. The reason for clip is simple: if the new policy changes too much relative to the old policy, PPO truncates the update to avoid destabilizing training in a single gradient step.

The PPO Workflow

Suppose the prompt is:

Write a Python function that checks whether a bracket string is valid.

The current policy samples this answer:

Scan the string with a stack; push left brackets, and when a right bracket appears, check whether it matches the stack top.

Training then goes through these steps:

A numeric example:

reward_model_score = 0.86
kl_penalty = 0.06
actual_return = 0.80

critic prediction at this prefix = 0.55
advantage = 0.80 - 0.55 = +0.25

This means the generation was better than the critic expected, so PPO increases the probability of the relevant tokens in that answer. But if a token probability changes too aggressively:

ratio = pi_theta(token) / pi_old(token) = 1.35
clip range = [0.8, 1.2]

the surrogate objective for that sample is clipped to 1.2 * advantage, so it no longer encourages the ratio to grow further. The intuition is:

Move toward higher reward, but do not take too large a step at once.

If another answer has low reward:

actual_return = 0.20
critic prediction = 0.55
advantage = -0.35

the corresponding token probabilities are pushed down. PPO is not just asking “what is the reward?” It is asking:

Was this generation better than the critic expected?
Did the new policy change too much relative to the old policy?
Is the new policy too far from the reference model?

PPO strengths:

• It is online RL and can directly optimize a reward signal.
• It is suitable for general RLHF when the reward model is reasonably reliable.
• With a critic and clipping, it is more stable than raw REINFORCE.

PPO weaknesses:

• It is engineering-heavy: policy / reference / reward / critic models must work together.
• Unstable critic training directly corrupts the advantage estimates.
• It is sensitive to hyperparameters, KL coefficients, and reward scale.

3. DPO: Preference Optimization, Not Traditional Online RL

DPO directly consumes preference pairs:

prompt x
chosen answer y_w > rejected answer y_l

It does not require training an explicit reward model first, and it does not need online rollout. The core idea is to make the model prefer the chosen answer over the rejected answer relative to a reference model:

log pi_theta(y_w | x) - log pi_ref(y_w | x)
should be larger than
log pi_theta(y_l | x) - log pi_ref(y_l | x)

An intuitive DPO loss:

loss = -log sigmoid(
  beta * [
    log_ratio(chosen) - log_ratio(rejected)
  ]
)

where log_ratio(y)=log pi_theta(y|x)-log pi_ref(y|x), and beta controls preference strength and conservatism relative to the reference. The reference model is still crucial: it is not a critic; it is the behavioral anchor that limits how far the model moves away from the original policy.

Technically, DPO is best described as an offline preference optimization method derived from KL-regularized RLHF. It uses the closed-form relationship between an optimal KL-regularized policy and its implicit reward to avoid training a separate reward model and running online RL.

How DPO Works

DPO does not let the model learn by online trial and error. It trains directly on preference data. For example:

Prompt:
Explain the difference between a reward model and a critic.

Chosen:
A reward model is a judge that scores a complete answer; a critic is a predictor that estimates how much future reward the current state can obtain.

Rejected:
A reward model and a critic both score the model; there is not much difference.

DPO asks whether the current model, relative to the reference model, prefers the chosen answer.

Suppose the current policy log probabilities are:

log pi_theta(chosen | x)   = -120
log pi_theta(rejected | x) = -115

This means the current model is actually more likely to generate the rejected answer, because -115 is a higher log probability than -120.

Now look at the reference model:

log pi_ref(chosen | x)   = -118
log pi_ref(rejected | x) = -116

DPO compares the two reference-relative ratios:

chosen_log_ratio   = -120 - (-118) = -2
rejected_log_ratio = -115 - (-116) = +1
gap = chosen_log_ratio - rejected_log_ratio = -3

The gap is negative, so the current model is more tilted toward the rejected answer than the reference is. The DPO loss pushes it to:

increase log pi_theta(chosen | x)
decrease log pi_theta(rejected | x)

So the training signal is not:

this answer has reward = 0.8

It is:

for the same prompt, the chosen answer should improve relative to the reference more than the rejected answer

In practice, implementations usually sum or average token log probabilities across the whole answer. Answer length, truncation, and tokenizer behavior all affect the values, so chosen / rejected samples should be comparable under the same prompt.

DPO’s main advantage is engineering simplicity: no rollout, no explicit / separate online reward-model scoring, and no critic. Its limitation comes from the same place: the model learns only from existing preference pairs and does not actively explore new, better answers.

DPO strengths:

• Simple and stable; no online RL environment is required.
• No separate reward model or critic is needed.
• Very convenient when chosen / rejected preference pairs already exist.

DPO weaknesses:

• It does not actively explore new answers; it learns from offline preference data.
• For math and code tasks with verifiable outcomes, direct online RLVR is often stronger.
• If preference coverage is poor, the model may learn style more than capability.

4. GRPO: Critic-Free RL Often Used for Reasoning Models

The core idea of GRPO is to sample multiple answers for the same prompt and compare them within the group.

Prompt x
├── answer A -> reward 0.9
├── answer B -> reward 0.7
├── answer C -> reward 0.2
└── answer D -> reward 0.0

Use the group mean as the baseline and the group standard deviation for normalization:

mean = 0.45
std  = 0.36
A_adv = (0.9 - 0.45) / 0.36 = +1.25
B_adv = (0.7 - 0.45) / 0.36 = +0.69
C_adv = (0.2 - 0.45) / 0.36 = -0.69
D_adv = (0.0 - 0.45) / 0.36 = -1.25

This lets GRPO remove the value model / critic:

PPO:  reward function + critic baseline
GRPO: reward function + group-relative baseline

How GRPO Works

Suppose the prompt is a math problem:

If 3x + 5 = 20, what is x?

GRPO does not sample only one answer. It samples a group of answers for the same prompt:

The group mean reward is the baseline, and the group standard deviation normalizes scale:

mean_reward = (1.0 + 0.0 + 1.0 + 0.0) / 4 = 0.5
std_reward  = 0.5

Each answer’s advantage can be understood as:

A_adv = (1.0 - 0.5) / 0.5 = +1.0
B_adv = (0.0 - 0.5) / 0.5 = -1.0
C_adv = (1.0 - 0.5) / 0.5 = +1.0
D_adv = (0.0 - 0.5) / 0.5 = -1.0

The update is intuitive:

increase the token probabilities for A / C
decrease the token probabilities for B / D

There is no separate value model. The baseline comes from the group mean for the same prompt; the standard deviation only rescales the advantage. In other words:

PPO asks the critic: roughly how much future reward is this prefix worth?
GRPO asks the group: is this answer better or worse than the other answers for the same problem?

GRPO still usually keeps PPO-like ratio / KL control so the model does not over-optimize reward and drift too far from the reference policy. It is especially suitable for math, code, multiple-choice, and format-verifiable tasks because rewards can be computed automatically:

math: whether the final answer is correct
code: whether unit tests pass
tool use: whether the task is completed and the evidence satisfies the requirement

GRPO is especially natural for RLVR, or reinforcement learning with verifiable rewards:

• Math: the final answer can be checked.
• Code: unit tests can verify behavior.
• STEM / reasoning: a rule-based verifier or judge can be used; if using an LLM-as-judge, watch for judge bias and reward hacking.

GRPO strengths:

• No critic training, so memory and engineering complexity are lower.
• Multiple samples for the same prompt naturally provide relative comparison signals.
• DeepSeekMath introduced GRPO; DeepSeek-R1-Zero used GRPO-style large-scale RL; DeepSeek-R1 then added cold-start SFT, rejection sampling / SFT, and additional RL stages, so R1 should not be simplified to “only GRPO.”

GRPO weaknesses:

• It still requires online rollout, so sampling cost is high.
• Rewards should preferably be verifiable; purely subjective preference tasks are not always a good fit.
• Group size, reward normalization, and KL control all affect stability.

5. Do Not Confuse Reward Model and Critic

The most interview-relevant distinction:

reward model: answer -> score
critic: partial trajectory/state -> expected return
advantage: per-token/step return estimate (often GAE / KL-shaped) - critic prediction

6. How to Choose

The most practical interview answer:

PPO is the traditional RLHF workhorse: complete but heavy. DPO is the preference-alignment workhorse: offline, simple, and stable. GRPO is a reasoning-RL workhorse: it replaces the critic with group-relative reward and is well suited to verifiable math/code-style tasks.

7. Common Questions

Is DPO RL?

In a broad sense, it is derived from an RLHF objective. In the narrower, traditional sense, it is not online RL: there is no rollout, no environment interaction, and no critic. It is more accurate to call it offline preference optimization.

Why Does GRPO Not Need a Critic?

Because it samples multiple answers for the same prompt, uses the group reward mean as the baseline, and uses the group standard deviation to normalize advantage. The critic’s role is to provide a baseline; GRPO replaces it with a group baseline.

What Is the Biggest Difference Between PPO and GRPO?

PPO usually trains a value model / critic to estimate advantage. GRPO estimates advantage from relative rewards among multiple samples for the same prompt. PPO is more general; GRPO is cheaper and more direct for verifiable reasoning tasks.

What Do All Three Share?

All three control how much the new model’s probabilities change relative to an old policy or reference model. PPO uses ratio clipping and KL, DPO uses a reference-relative preference loss, and GRPO usually keeps KL or ratio clipping as well.

8. Final Cheat Sheet

PPO:
  online RLHF, reward model/function + critic + KL/clip
  strongest general RLHF baseline, but expensive and sensitive

DPO:
  offline preference optimization, chosen/rejected pair
  no rollout, no explicit reward model, no critic
  simple, stable, good for alignment

GRPO:
  online critic-free RL, group-relative advantage
  no value model, good for verifiable reasoning rewards
  common in modern verifiable reasoning RL