9 papers
Your Self-Play Algorithm is Secretly an Adversarial Imitator: Understanding LLM Self-Play through the Lens of Imitation Learning
Shangzhe Li, Xuchao Zhang, Chetan Bansal +1
Self-play post-training methods has emerged as an effective approach for finetuning large language models and turn the weak language model into strong language model without prefer…
OGLS-SD: On-Policy Self-Distillation with Outcome-Guided Logit Steering for LLM Reasoning
Yuxiao Yang, Xiaoyun Wang, Weitong Zhang
We study on-policy self-distillation (OPSD), where a language model improves its reasoning ability by distilling privileged teacher distributions along its own on-policy trajectori…
LARK: Learnability-Grounded Trajectory Selection for Efficient Reasoning Distillation
Tianrun Yu, Kaixiang Zhao, Chih-Chun Chen +5
We study trajectory selection for reasoning distillation, where teacher-generated reasoning trajectories are selectively used as supervision for a student model. Existing methods r…
Moment Matching Q-Learning
Yiyan, Liang, Sifei Liu +1
Score-based and flow-based generative models exhibit remarkable expressive capacity in capturing complex distributions, and have been extensively deployed in tasks ranging from ima…
Return-to-Go Is More Than a Number: Q-Guided Alignment for Return-Conditioned Supervised Learning
Yuxiao Yang, Weitong Zhang
Conditioned Sequence Models (CSMs) learn policies by treating return-to-go (RTG) as a control signal. However, existing CSMs often treat the RTGs as simple numerical inputs rather…
Provably Efficient Offline-to-Online Value Adaptation with General Function Approximation
Shangzhe Li, Weitong Zhang
We study value adaptation in offline-to-online reinforcement learning under general function approximation. Starting from an imperfect offline pretrained -function, the learner…