GRPO, DPO, and RLVR: Post-Training Explained Without the Notation

If you read a model card for a reasoning model, you meet three acronyms in the first paragraph: SFT you probably know; DPO, GRPO, and RLVR you may be nodding along to. They are simpler than the papers make them look, and understanding them tells you something practical: which tasks these models will be good at, and why.

Where They Fit

A base model predicts the next token. Post-training makes it useful, in stages:

1. SFT (supervised fine-tuning). Imitate examples of good responses.
2. Preference optimization (DPO). Prefer better responses over worse ones.
3. RL with verifiable rewards (RLVR, using GRPO). Get better than the examples on tasks where correctness can be checked.

Stage three is where reasoning comes from.

DPO in One Idea

You have pairs: a prompt, a response humans preferred, and one they rejected. You want the model to assign higher probability to the preferred one. The old way (RLHF with PPO) trained a separate reward model on the pairs, then used reinforcement learning to push the policy toward high reward. Direct Preference Optimization skips the reward model: it derives a loss directly from the pairs that has the same effect.

pythonCopy
def dpo_loss(policy, ref, prompt, chosen, rejected, beta=0.1):
    # log-ratio of how much more the policy likes each response than the reference does
    chosen_lr = policy.logprob(prompt, chosen) - ref.logprob(prompt, chosen)
    rejected_lr = policy.logprob(prompt, rejected) - ref.logprob(prompt, rejected)
    return -log_sigmoid(beta * (chosen_lr - rejected_lr))

Increase the gap between chosen and rejected, relative to a frozen reference so the model does not drift. Cheap, stable, and the default for style, helpfulness, and safety tuning. Its limit: it can only teach preferences you have labeled, and it cannot make the model better than the best response in the data.

Verifiable Rewards in One Idea

For some tasks you do not need a human to say which response is better. You can check: the numeric answer matches, the unit tests pass, the output parses against a schema, the proof verifies. That check is a verifiable reward. RLVR is reinforcement learning where the reward comes from such a checker.

Why it matters: the reward is free at scale, and it does not cap the model at the quality of its examples. The model can sample, get checked, and improve on whatever it discovers. Because nothing constrains how it gets to the answer, it learns to spend tokens on intermediate steps, backtracking, and self-checking, because those raise the reward. That is what a reasoning model is.

The constraint is the verifier. Math, code, structured extraction, logic, games: verifiable. "Write a compelling email": not. That is why reasoning models' gains cluster where they do.

GRPO in One Idea

Classic PPO needs a value model to estimate how good a partial response is, doubling the memory and adding instability. Group Relative Policy Optimization drops it. For each prompt, sample a group of G responses, score each with the verifier, and use each response's reward relative to the group's average as its advantage.

pythonCopy
def grpo_step(policy, ref, prompt, verify, G=8, kl_coef=0.04):
    responses = [policy.sample(prompt) for _ in range(G)]
    rewards = [verify(prompt, r) for r in responses]        # e.g. 1.0 if tests pass else 0.0
    mu = sum(rewards) / G
    sigma = (sum((r - mu) ** 2 for r in rewards) / G) ** 0.5 + 1e-6
    advantages = [(r - mu) / sigma for r in rewards]
    loss = 0.0
    for resp, adv in zip(responses, advantages):
        loss += -adv * policy.logprob(prompt, resp)            # push toward above-average responses
        loss += kl_coef * kl(policy, ref, prompt, resp)        # stay near the reference
    return loss / G

Group normalization is the trick. A response is "good" only relative to its siblings, so the signal stays informative whether the prompt is easy (most pass) or hard (most fail). The KL term keeps the model from finding degenerate outputs that satisfy the checker without being useful.

What This Means for You

• Reasoning is a training outcome. "Think step by step" in a prompt borrows the form, not the training.
• Verifiable tasks get the gains. Expect reasoning models to help most on math, code, planning with checkable outcomes, and structured output, and least on open-ended writing.
• You can run this. A small open-weight model, a narrow verifiable task, a few hundred verifier-filtered examples for SFT, and a short GRPO phase with the same verifier fits on one GPU. It is a serious portfolio project.
• Verifier design is the frontier. The more of a task you can make checkable, the more RL can improve it. That is an engineering skill, and it is the same skill as writing a good eval.

Common Questions

What is GRPO? Group Relative Policy Optimization is a reinforcement learning algorithm for post-training that samples a group of responses per prompt, scores them, and uses each response's reward relative to the group's mean as its training signal. It drops the separate value model that PPO needs, which makes it cheaper and more stable.

What is RLVR? Reinforcement learning with verifiable rewards: post-training where the reward is computed by a checker (tests pass, the numeric answer matches, the output parses) instead of a learned reward model or human labels. It is the dominant recipe for reasoning models because it scales without annotators.

When should I use DPO instead? When the task has no cheap verifier. Direct Preference Optimization trains directly on pairs of preferred and rejected responses and is the practical choice for style, helpfulness, and safety, where "correct" cannot be computed.

What to Practice Next

Pick a task with a checker: text-to-SQL for one schema, JSON extraction against a schema, a small math domain. Write the verifier first. Generate 500 candidates with a large model, keep the ones that pass, hold out 100, SFT a small model on 400, then run 200 GRPO steps with the same verifier and report whether it beats SFT on the held-out set. The module Reasoning Models, Post-Training, and Test-Time Compute covers the full pipeline and the cost side.