Why Tree-Style Branching Matters for Thought Advantage Estimation in GRPO

Hongcheng Wang, Yinuo Huang, Sukai Wang +2

Group Relative Policy Optimization (GRPO) trains Chain-of-Thought reasoning with verifiable rewards, but estimating thought-level advantages without value functions often suffers f…