6 papers
Trust Region Masking for Long-Horizon LLM Reinforcement Learning
Yingru Li, Jiacai Liu, Jiawei Xu +4
Policy gradient methods for Large Language Models optimize a policy via a surrogate objective computed from samples of a rollout policy . However, modern…
The Optimal Token Baseline: Variance Reduction for Long-Horizon LLM-RL
Yingru Li, Jiawei Xu, Ziniu Li +10
Reinforcement Learning (RL) for Large Language Models (LLMs) often suffers from training collapse in long-horizon tasks due to exploding gradient variance. To mitigate this, a base…
Dynamic Vocabulary Pruning: Stable LLM-RL by Taming the Tail
Yingru Li, Jiawei Xu, Jiacai Liu +6
Reinforcement Learning (RL) for Large Language Models (LLMs) faces a fundamental tension: the numerical divergence between high-throughput inference engines and numerically precise…
Scalable Exploration via Ensemble++
Yingru Li, Jiawei Xu, Baoxiang Wang +1
Thompson Sampling is a principled method for balancing exploration and exploitation, but its real-world adoption faces computational challenges in large-scale or non-conjugate sett…
ADG: Ambient Diffusion-Guided Dataset Recovery for Corruption-Robust Offline Reinforcement Learning
Zeyuan Liu, Zhihe Yang, Jiawei Xu +5
Real-world datasets collected from sensors or human inputs are prone to noise and errors, posing significant challenges for applying offline reinforcement learning (RL). While exis…
Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modeling
Jiawei Xu, Rui Yang, Shuang Qiu +4
Learning policy from offline datasets through offline reinforcement learning (RL) holds promise for scaling data-driven decision-making while avoiding unsafe and costly online inte…