A First-Principles Derivation of LLM Policy Optimization: From Expected Reward to GRPO and Its Structural Extensions

Jianghan Shen, Siqi Luo, Yue Li +9

Policy gradient algorithms for language models optimize the same objective $J(\theta )=\mathbb{E}\ast \tau \sim p\ast \theta (\tau )[R(\tau )]$, which has exactly two factors: the trajectory probability $p_θ(…