8 papers
Trust Region Masking for Long-Horizon LLM Reinforcement Learning
Yingru Li, Jiacai Liu, Jiawei Xu +4
Policy gradient methods for Large Language Models optimize a policy via a surrogate objective computed from samples of a rollout policy . However, modern…
The Optimal Token Baseline: Variance Reduction for Long-Horizon LLM-RL
Yingru Li, Jiawei Xu, Ziniu Li +10
Reinforcement Learning (RL) for Large Language Models (LLMs) often suffers from training collapse in long-horizon tasks due to exploding gradient variance. To mitigate this, a base…
Choosing How to Remember: Adaptive Memory Structures for LLM Agents
Mingfei Lu, Mengjia Wu, Feng Liu +8
Memory is critical for enabling large language model (LLM) based agents to maintain coherent behavior over long-horizon interactions. However, existing agent memory systems suffer…
Dr. Kernel: Reinforcement Learning Done Right for Triton Kernel Generations
Wei Liu, Jiawei Xu, Yingru Li +4
High-quality kernel is critical for scalable AI systems, and enabling LLMs to generate such code would advance AI development. However, training LLMs for this task requires suffici…
Dynamic Vocabulary Pruning: Stable LLM-RL by Taming the Tail
Yingru Li, Jiawei Xu, Jiacai Liu +6
Reinforcement Learning (RL) for Large Language Models (LLMs) faces a fundamental tension: the numerical divergence between high-throughput inference engines and numerically precise…
Beyond Precision: Training-Inference Mismatch is an Optimization Problem and Simple LR Scheduling Fixes It
Yaxiang Zhang, Yingru Li, Jiacai Liu +4
Reinforcement Learning (RL) for training Large Language Models is notoriously unstable. While recent studies attribute this to "training inference mismatch stemming" from inconsist…