8 papers
Privy to the Foil: Recasting Value Estimation with a Self-Privileged Critic for RLVR
Kun Liang, Chenming Tang, Clive Bai +5
Assigning credit to intermediate steps remains a central challenge in training Large Language Models (LLMs) on multi-step reasoning tasks with sparse terminal rewards, and actor-cr…
CAFE: Self-Improving Search Agents Need Co-Evolving Feedback
Boyang Liu, Senjie Jin, Peixin Wang +17
Reliable search requires more than acquiring external evidence. An agent must also recognize and recover from errors as its trajectory unfolds. In-trajectory feedback provides a me…
Learning Latent Reasoning Traces for Scalar Reward Models End-to-End
Sanwoo Lee, Clive Bai, Hsiu-Yuan Huang +3
Reward models (RMs) are central to aligning large language models with human preferences via reinforcement learning. Although traditional scalar RMs enable efficient and probabilis…
ADWIN: Adaptive Windows for Horizon-Aware On-Policy Distillation
Kun Liang, Chenming Tang, Clive Bai +3
On-policy distillation (OPD) transfers reasoning behavior by training a student on teacher feedback along student-generated trajectories, but standard full-rollout training ties ev…
Listwise Policy Optimization: Group-based RLVR as Target-Projection on the LLM Response Simplex
Yun Qu, Qi Wang, Yixiu Mao +11
Reinforcement learning with verifiable rewards (RLVR) has become a standard approach for large language models (LLMs) post-training to incentivize reasoning capacity. Among existin…
Composition-RL: Compose Your Verifiable Prompts for Reinforcement Learning of Large Language Models
Xin Xu, Clive Bai, Kai Yang +8
Large-scale verifiable prompts underpin the success of Reinforcement Learning with Verifiable Rewards (RLVR), but they contain many uninformative examples and are costly to expand…
Small Generalizable Prompt Predictive Models Can Steer Efficient RL Post-Training of Large Reasoning Models
Yun Qu, Qi Wang, Yixiu Mao +8
Reinforcement learning enhances the reasoning capabilities of large language models but often involves high computational costs due to rollout-intensive optimization. Online prompt…
ORBIT: On-policy Exploration-Exploitation for Controllable Multi-Budget Reasoning
Kun Liang, Clive Bai, Xin Xu +5
Recent Large Reasoning Models (LRMs) achieve strong performance by leveraging long-form Chain-of-Thought (CoT) reasoning, but uniformly applying overlong reasoning at inference tim…