9 papers
On the Impossibility of Unbiased and Length-Invariant Policy Optimization with Outcome Rewards
Fei Ding, Yongkang Zhang, Yuhao Liao +2
Group Relative Policy Optimization (GRPO) is the dominant reinforcement learning algorithm for training reasoning capabilities in large language models, notably adopted by DeepSeek…
ThinkReset: Learnable Intermediate Interface Construction for Bounded-Context Long-Horizon Reasoning
Fei Ding, Yongkang Zhang, Runhao Liu +2
Long chain-of-thought reasoning improves performance on complex problems, but it also introduces redundancy accumulation, context overflow, and error anchoring. We argue that under…
Internalizing Outcome Supervision into Process Supervision: A New Paradigm for Reinforcement Learning for Reasoning
Fei Ding, Yongkang Zhang, Runhao Liu +4
The central challenge of reinforcement learning for reasoning lies not only in the sparsity of outcome-level supervision, but more fundamentally in how to transform feedback provid…
Rethinking the Comparison Unit in Sequence-Level Reinforcement Learning: An Equal-Length Paired Training Framework from Loss Correction to Sample Construction
Fei Ding, Yongkang Zhang, Runhao Liu +5
This paper investigates the length problem in sequence-level relative reinforcement learning. We observe that, although existing methods partially alleviate length-related phenomen…
Scaffold-Mediated Post-Training: Co-Evolving Model Parameters and Procedural Scaffold Graphs
Fei Ding, Yongkang Zhang, Runhao Liu +3
Post-training of large language models optimizes only parameters, while inference-time procedural scaffolds are typically designed independently of parameter training. This disconn…
State commitment learning: training language models to distinguish computation from memory
Fei Ding, Yongkang Zhang, Runhao Liu +3
Reasoning language models do not distinguish tokens used for computation from tokens that constitute persistent state: once generated, all hidden thoughts remain in context and inf…