Glossary › DPO
DPO (Direct Preference Optimization)
A post-training method that trains a model directly on pairs of (preferred, rejected) responses without a separate reward model. Simpler and cheaper than RLHF with PPO; the standard first step when you have preference data.
Where DPO shows up
All matching postsReasoning Models, Post-Training, and Test-Time Compute
Where reasoning models come from, what they cost, and when to pay for them. The SFT to DPO to GRPO pipeline, reinforcement learning with verifiable rewards, test-time compute as a dial, and a method for deciding which of your requests deserve a thinking model.
Fine-Tuning and Post-Training: LoRA, SFT, DPO, and Reasoning RL
What actually happens after pretraining, and when you should do any of it yourself. Parameter-efficient fine-tuning with LoRA, supervised fine-tuning data, preference optimization, and the reinforcement learning recipe behind reasoning models, with a decision framework and a project you can run on one GPU.
GRPO, DPO, and RLVR: Post-Training Explained Without the Notation
Three acronyms show up in every reasoning-model paper. Here is what each one does, why GRPO replaced PPO for most teams, what a verifiable reward is and why it is the whole constraint, and short runnable code for each idea.