8 papers
DPO Unchained: Your Training Algorithm is Secretly Disentangled in Human Choice Theory (and its Loss' Convexity is Dispensable)
Wenxuan Zhou, Shujian Zhang, Brice Magdalou +4
Normative theories allow one to elicit key parts of a ML algorithm from first principles, which is crucial at a time of championed scrutiny for ML work. Direct Preference Optimizat…
SFTMix: Elevating Language Model Instruction Tuning with Mixup Recipe
Yuxin Xiao, Shujian Zhang, Wenxuan Zhou +2
To acquire instruction-following capabilities, large language models (LLMs) undergo instruction tuning, where they are trained on instruction-response pairs using next-token predic…
Steering LLMs for Culturally Localized Generation
Simran Khanuja, Hongbin Liu, Shujian Zhang +4
LLMs are deployed globally, yet produce responses biased towards cultures with abundant training data. Existing cultural localization approaches such as prompting or post-training…
MUSIC: MUlti-Step Instruction Contrast for Multi-Turn Reward Models
Wenzhe Li, Shujian Zhang, Wenxuan Zhou +5
Evaluating the quality of multi-turn conversations is crucial for developing capable Large Language Models (LLMs), yet remains a significant challenge, often requiring costly human…
Fantastic Reasoning Behaviors and Where to Find Them: Unsupervised Discovery of the Reasoning Process
Zhenyu Zhang, Shujian Zhang, John Lambert +6
Despite the growing reasoning capabilities of recent large language models (LLMs), their internal mechanisms during the reasoning process remain underexplored. Prior approaches oft…
Eliciting Behaviors in Multi-Turn Conversations
Jing Huang, Shujian Zhang, Lun Wang +3
Identifying specific and often complex behaviors from large language models (LLMs) in conversational settings is crucial for their evaluation. Recent work proposes novel techniques…