GRPO vs PPO vs DPO on GSM8K: What I Learned Building RL Training from Scratch (opens in new tab)

I implemented three RL algorithms from scratch, ran a controlled comparison on AWS, and found something I didn’t expect. Here’s what…