14 papers
Scaffold-Mediated Post-Training: Co-Evolving Model Parameters and Procedural Scaffold Graphs
Fei Ding, Yongkang Zhang, Runhao Liu +3
Post-training of large language models optimizes only parameters, while inference-time procedural scaffolds are typically designed independently of parameter training. This disconn…
On the Impossibility of Unbiased and Length-Invariant Policy Optimization with Outcome Rewards
Fei Ding, Yongkang Zhang, Yuhao Liao +2
Group Relative Policy Optimization (GRPO) is the dominant reinforcement learning algorithm for training reasoning capabilities in large language models, notably adopted by DeepSeek…
ThinkReset: Learnable Intermediate Interface Construction for Bounded-Context Long-Horizon Reasoning
Fei Ding, Yongkang Zhang, Runhao Liu +2
Long chain-of-thought reasoning improves performance on complex problems, but it also introduces redundancy accumulation, context overflow, and error anchoring. We argue that under…
Reducing Credit Assignment Variance via Counterfactual Reasoning Paths
Fei Ding, Yongkang Zhang, Youwei Wang +1
Reinforcement learning for multi-step reasoning with large language models (LLMs) typically relies on sparse terminal rewards, which creates a poorly conditioned credit-assignment…
Internalizing Outcome Supervision into Process Supervision: A New Paradigm for Reinforcement Learning for Reasoning
Fei Ding, Yongkang Zhang, Runhao Liu +4
The central challenge of reinforcement learning for reasoning lies not only in the sparsity of outcome-level supervision, but more fundamentally in how to transform feedback provid…
Rethinking the Comparison Unit in Sequence-Level Reinforcement Learning: An Equal-Length Paired Training Framework from Loss Correction to Sample Construction
Fei Ding, Yongkang Zhang, Runhao Liu +5
This paper investigates the length problem in sequence-level relative reinforcement learning. We observe that, although existing methods partially alleviate length-related phenomen…
Design Conditions for Intra-Group Learning of Sequence-Level Rewards: Token Gradient Cancellation
Fei Ding, Yongkang Zhang, youwei wang +1
Reinforcement learning for multi-step reasoning with large language models (LLMs) typically relies on sparse terminal rewards, which creates a poorly conditioned credit-assignment…
State commitment learning: training language models to distinguish computation from memory
Fei Ding, Yongkang Zhang, Runhao Liu +3
Reasoning language models do not distinguish tokens used for computation from tokens that constitute persistent state: once generated, all hidden thoughts remain in context and inf…
DexSim2Real: Foundation Model-Guided Sim-to-Real Transfer for Generalizable Dexterous Manipulation
Zijian Zeng, Fei Ding, Huiming Yang +2
Sim-to-real transfer remains a critical bottleneck for deploying dexterous manipulation policies learned in simulation to real-world robots. Existing approaches rely on manually de…
HELM: Harness-Enhanced Long-horizon Memory for Vision-Language-Action Manipulation
Zijian Zeng, Fei Ding, Huiming Yang +1
Vision-Language-Action (VLA) models fail systematically on long-horizon manipulation tasks despite strong short-horizon performance. We show that this failure is not resolved by ex…
Difference Feedback: Generating Multimodal Process-Level Supervision for VLM Reinforcement Learning
Feiding, Yongkang Zhang, Yuhao Liao +10
Vision--language models (VLMs) are increasingly aligned via Group Relative Policy Optimization (GRPO)-style training. However, relying solely on terminal outcome rewards yields spa…
PoseStreamer: A Multi-modal Framework for 3D Tracking of Unseen Moving Objects
Huiming Yang, Linglin Liao, Fei Ding +2
Six degree of freedom (6DoF) pose estimation for novel objects is a critical task in computer vision, yet it faces significant challenges in high-speed and low-light scenarios wher…
Practice on Long Behavior Sequence Modeling in Tencent Advertising
Xian Hu, Ming Yue, Zhixiang Feng +24
Long-sequence modeling has become an indispensable frontier in recommendation systems for capturing users' long-term preferences. However, user behaviors within advertising domains…
Multi-Layer GRPO: Enhancing Reasoning and Self-Correction in Large Language Models
Fei Ding, Baiqiao Wang, Zijian Zeng +1
The Group Relative Policy Optimization (GRPO) algorithm has demonstrated considerable success in enhancing the reasoning capabilities of large language models (LLMs), as evidenced…