LLMs Can Learn to Reason Via Off-Policy RL

Daniel Ritter, Owen Oertell, Bradley Guo +3

Reinforcement learning (RL) approaches for Large Language Models (LLMs) frequently use on-policy algorithms, such as PPO or GRPO. However, policy lag from distributed training arch…