6 papers
Smaller Models, Better Rejects: Preference Distillation Scaling
Rui Cai, Wenhui Zhu, Xiwen Chen +15
Preference distillation typically treats a teacher response as preferred and the student's own response as rejected. This assumes that self-generated failures are the most informat…
Rethinking Personalized Generation: Test-Time Alignment via Factorized Ranking Models
Qiyao Ma, Junshan Zhang, Zhe Zhao
Aligning large language models (LLMs) to diverse user preferences is fundamentally hindered by standard alignment paradigms that optimize for monolithic users. In this work, empiri…
ModelLens: Finding the Best for Your Task from Myriads of Models
Rui Cai, Weijie Jacky Mo, Xiaofei Wen +5
The open-source model ecosystem now contains hundreds of thousands of pretrained models, yet picking the best model for a new dataset is increasingly infeasible: new models and unb…
Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization
Qiyao Ma, Dechen Gao, Rui Cai +4
Pluralistic alignment has emerged as a critical frontier in the development of Large Language Models (LLMs), with reward models (RMs) serving as a central mechanism for capturing d…
From Faithfulness to Correctness: Generative Reward Models that Think Critically
Qiyao Ma, Yunsheng Shi, Hongtao Tian +3
Through reinforcement learning with verifiable rewards (RLVR), large language models have achieved substantial progress in domains with easily verifiable outcomes, such as mathemat…
Learning More with Less: A Dynamic Dual-Level Down-Sampling Framework for Efficient Policy Optimization
Chao Wang, Tao Yang, Hongtao Tian +5
Critic-free methods like GRPO reduce memory demands by estimating advantages from multiple rollouts but tend to converge slowly, as critical learning signals are diluted by an abun…