11 papers
Start Classifying: Categorical Critics for LLM Reinforcement Learning
Zhijian Zhou, Long Li, Xuan Zhang +7
Proximal Policy Optimization (PPO) for large language models typically trains its critic by mean-squared-error (MSE) regression on scalar value targets. Although scalar MSE is stat…
SPPO: Sequence-Level PPO for Long-Horizon Reasoning Tasks
Tianyi Wang, Yixia Li, Long Li +6
Proximal Policy Optimization (PPO) is central to aligning Large Language Models (LLMs) in reasoning tasks with verifiable rewards. However, standard token-level PPO struggles in th…
SQL-ASTRA: Alleviating Sparse Feedback in Agentic SQL via Column-Set Matching and Trajectory Aggregation
Long Li, Zhijian Zhou, Jiangxuan Long +5
Agentic Reinforcement Learning (RL) shows promise for complex tasks, but Text-to-SQL remains mostly restricted to single-turn paradigms. A primary bottleneck is the credit assignme…
DyJR: Preserving Diversity in Reinforcement Learning with Verifiable Rewards via Dynamic Jensen-Shannon Replay
Long Li, Zhijian Zhou, Tianyi Wang +7
While Reinforcement Learning (RL) enhances Large Language Model reasoning, on-policy algorithms like GRPO are sample-inefficient as they discard past rollouts. Existing experience…
The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Reward
Long Li, Zhijian Zhou, Jiaran Hao +9
A central paradox in fine-tuning Large Language Models (LLMs) with Reinforcement Learning with Verifiable Reward (RLVR) is the frequent degradation of multi-attempt performance (Pa…
Anchored Policy Optimization: Mitigating Exploration Collapse Via Support-Constrained Rectification
Tianyi Wang, Long Li, Hongcan Guo +5
Reinforcement Learning with Verifiable Rewards (RLVR) is increasingly viewed as a tree pruning mechanism. However, we identify a systemic pathology termed Recursive Space Contracti…