The Optimal Token Baseline: Variance Reduction for Long-Horizon LLM-RL

Yingru Li, Jiawei Xu, Ziniu Li +10

Reinforcement Learning (RL) for Large Language Models (LLMs) often suffers from training collapse in long-horizon tasks due to exploding gradient variance. To mitigate this, a base…