7 papers · 1 filter
The Optimal Token Baseline: Variance Reduction for Long-Horizon LLM-RL
Yingru Li, Jiawei Xu, Ziniu Li +10
Reinforcement Learning (RL) for Large Language Models (LLMs) often suffers from training collapse in long-horizon tasks due to exploding gradient variance. To mitigate this, a base…
Dynamic Vocabulary Pruning: Stable LLM-RL by Taming the Tail
Yingru Li, Jiawei Xu, Jiacai Liu +6
Reinforcement Learning (RL) for Large Language Models (LLMs) faces a fundamental tension: the numerical divergence between high-throughput inference engines and numerically precise…
Trust Region Masking for Long-Horizon LLM Reinforcement Learning
Yingru Li, Jiacai Liu, Jiawei Xu +4
Policy gradient methods for Large Language Models optimize a policy via a surrogate objective computed from samples of a rollout policy . However, modern LLM…
ADG: Ambient Diffusion-Guided Dataset Recovery for Corruption-Robust Offline Reinforcement Learning
Zeyuan Liu, Zhihe Yang, Jiawei Xu +5
Real-world datasets collected from sensors or human inputs are prone to noise and errors, posing significant challenges for applying offline reinforcement learning (RL). While exis…
Scalable Exploration via Ensemble++
Yingru Li, Jiawei Xu, Baoxiang Wang +1
Thompson Sampling is a principled method for balancing exploration and exploitation, but its real-world adoption faces computational challenges in large-scale or non-conjugate sett…
Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modeling
Jiawei Xu, Rui Yang, Shuang Qiu +4
Learning policy from offline datasets through offline reinforcement learning (RL) holds promise for scaling data-driven decision-making while avoiding unsafe and costly online inte…
Relative Policy-Transition Optimization for Fast Policy Transfer
Jiawei Xu, Cheng Zhou, Yizheng Zhang +2
We consider the problem of policy transfer between two Markov Decision Processes (MDPs). We introduce a lemma based on existing theoretical results in reinforcement learning to mea…