9 papers
Unbiased Top- Estimation for On-Policy Distillation
Linjian Meng, Siyuan Gan, YuHan Li +5
On-policy distillation (OPD) is becoming an important component of large language model (LLM) post-training for transferring the reasoning capability of a strong teacher LLM to a w…
Learning from Hard Prompts: Difficulty-aware Advantage Amplification in Dynamic Sampling
Siyuan Gan, Yuhan Li, Xiran Wang +6
Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO) is a prominent variant of Group Relative Policy Optimization (GRPO). DAPO introduces several improvements over GRPO.…
When Teacher Guidance Misleads: Reward-Aligned On-Policy Distillation
Siyuan Gan, Yuhan Li, Xiran Wang +5
On-policy distillation (OPD) has recently emerged as a popular post-training paradigm for large language models (LLMs), providing an efficient way to transfer the knowledge and cap…
CoLVR: Enhancing Exploratory Latent Visual Reasoning via Contrastive Optimization
Ziyang Ding, Linjian Meng, Yiming Wu +3
Due to the potential for exploratory reasoning of Latent Visual Reasoning, recent works tend to enable MLLMs (Multimodal Large Language Models) to perform visual reasoning by propa…
Thinking-Based Non-Thinking: Solving the Reward Hacking Problem in Training Hybrid Reasoning Models via Reinforcement Learning
Siyuan Gan, Jiaheng Liu, Boyan Wang +8
Large reasoning models (LRMs) have attracted much attention due to their exceptional performance. However, their performance mainly stems from thinking, a long Chain of Thought (Co…
Tree-Based Stochastic Optimization for Solving Large-Scale Urban Network Security Games
Shuxin Zhuang, Linjian Meng, Shuxin Li +2
Urban Network Security Games (UNSGs), which model the strategic allocation of limited security resources on city road networks, are critical for urban safety. However, finding a Na…
Faster Game Solving via Asymmetry of Step Sizes
Linjian Meng, Tianpei Yang, Youzhi Zhang +2
Counterfactual Regret Minimization (CFR) algorithms are widely used to compute a Nash equilibrium (NE) in two-player zero-sum imperfect-information extensive-form games (IIGs). Amo…
Magnetic Preference Optimization: Achieving Last-iterate Convergence for Language Model Alignment
Mingzhi Wang, Chengdong Ma, Qizhi Chen +7
Self-play methods have demonstrated remarkable success in enhancing model capabilities across various domains. In the context of Reinforcement Learning from Human Feedback (RLHF),…
Efficient Last-iterate Convergence Algorithms in Solving Games
Linjian Meng, Youzhi Zhang, Zhenxing Ge +6
To establish last-iterate convergence for Counterfactual Regret Minimization (CFR) algorithms in learning a Nash equilibrium (NE) of extensive-form games (EFGs), recent studies ref…