5 papers · 1 filter
Geometric Self-Distillation for Reasoning Generalization
Josip JukiÄ, Ivan Titov
On-policy distillation is a practical post-training recipe for large language models, supplying dense teacher supervision on the student's own trajectories. In privileged-context s…
Blending Supervised and Reinforcement Fine-Tuning with Prefix Sampling
Zeyu Huang, Tianhao Cheng, Zihan Qiu +4
Existing LLMs-post-training techniques are broadly categorized into supervised fine-tuning (SFT) and reinforcement fine-tuning (RFT). Each paradigm presents a distinct trade-off: (…
The Cancellation Hypothesis in Critic-Free RL: From Outcome Rewards to Token Credits
Tianhao Cheng, Zeyu Huang, Zihan Qiu +5
A commonly accepted explanation of critic-free RL for LLMs, based on sequence-level rewards, is that it reinforces successful rollouts with a positive advantage while penalizing fa…
Scalpel vs. Hammer: GRPO Amplifies Existing Capabilities, SFT Replaces Them
Neel Rajani, Aryo Pradipta Gema, Seraphina Goldfarb-Tarrant +1
Training large language models (LLMs) for reasoning via maths and code datasets has become a major new focus in LLM post-training. Two particularly popular approaches are reinforce…
Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models
Zihan Qiu, Zeyu Huang, Bo Zheng +7
This paper revisits the implementation of oad-alancing oss (LBL) when training Mixture-of-Experts (MoEs) models. Specifically, LBL for MoEs is d…