10 papers
On the Impossibility of Unbiased and Length-Invariant Policy Optimization with Outcome Rewards
Fei Ding, Yongkang Zhang, Yuhao Liao +2
Group Relative Policy Optimization (GRPO) is the dominant reinforcement learning algorithm for training reasoning capabilities in large language models, notably adopted by DeepSeek…
ThinkReset: Learnable Intermediate Interface Construction for Bounded-Context Long-Horizon Reasoning
Fei Ding, Yongkang Zhang, Runhao Liu +2
Long chain-of-thought reasoning improves performance on complex problems, but it also introduces redundancy accumulation, context overflow, and error anchoring. We argue that under…
Reducing Credit Assignment Variance via Counterfactual Reasoning Paths
Fei Ding, Yongkang Zhang, Youwei Wang +1
Reinforcement learning for multi-step reasoning with large language models (LLMs) typically relies on sparse terminal rewards, which creates a poorly conditioned credit-assignment…
Internalizing Outcome Supervision into Process Supervision: A New Paradigm for Reinforcement Learning for Reasoning
Fei Ding, Yongkang Zhang, Runhao Liu +4
The central challenge of reinforcement learning for reasoning lies not only in the sparsity of outcome-level supervision, but more fundamentally in how to transform feedback provid…
Rethinking the Comparison Unit in Sequence-Level Reinforcement Learning: An Equal-Length Paired Training Framework from Loss Correction to Sample Construction
Fei Ding, Yongkang Zhang, Runhao Liu +5
This paper investigates the length problem in sequence-level relative reinforcement learning. We observe that, although existing methods partially alleviate length-related phenomen…
Design Conditions for Intra-Group Learning of Sequence-Level Rewards: Token Gradient Cancellation
Fei Ding, Yongkang Zhang, youwei wang +1
Reinforcement learning for multi-step reasoning with large language models (LLMs) typically relies on sparse terminal rewards, which creates a poorly conditioned credit-assignment…