7 papers
Beyond Trajectory Imitation: Strategy-Guided Policy Optimization for LLM Reasoning
Tianyuan Shi, Canbin Huang, Bei Li +4
Distilling reasoning capabilities from strong to weak language models typically involves imitating specific solution trajectories, effectively transferring what to answer rather th…
When Model Merging Breaks Routing: Training-Free Calibration for MoE
Canbin Huang, Tianyuan Shi, Xiaojun Quan +3
Model merging has emerged as a cost-effective approach for consolidating the capabilities of multiple LLMs without retraining. However, existing merging techniques, largely based o…
ProFuser: Progressive Fusion of Large Language Models
Tianyuan Shi, Fanqi Wan, Canbin Huang +6
While fusing the capacities and advantages of various large language models offers a pathway to construct more powerful and versatile models, a fundamental challenge is to properly…
Lookahead Routing for Large Language Models
Canbin Huang, Tianyuan Shi, Yuhua Zhu +2
Large language model (LLM) routers improve the efficiency of multi-model systems by directing each query to the most appropriate model while leveraging the diverse strengths of het…
Mutual-Taught for Co-adapting Policy and Reward Models
Tianyuan Shi, Canbin Huang, Fanqi Wan +5
During the preference optimization of large language models (LLMs), distribution shifts may arise between newly generated model samples and the data used to train the reward model…
FuseRL: Dense Preference Optimization for Heterogeneous Model Fusion
Longguang Zhong, Fanqi Wan, Ziyi Yang +3
Heterogeneous model fusion enhances the performance of LLMs by integrating the knowledge and capabilities of multiple structurally diverse models. However, existing approaches ofte…