4 papers
DPO Unchained: Your Training Algorithm is Secretly Disentangled in Human Choice Theory (and its Loss' Convexity is Dispensable)
Wenxuan Zhou, Shujian Zhang, Brice Magdalou +4
Normative theories allow one to elicit key parts of a ML algorithm from first principles, which is crucial at a time of championed scrutiny for ML work. Direct Preference Optimizat…
MUSIC: MUlti-Step Instruction Contrast for Multi-Turn Reward Models
Wenzhe Li, Shujian Zhang, Wenxuan Zhou +5
Evaluating the quality of multi-turn conversations is crucial for developing capable Large Language Models (LLMs), yet remains a significant challenge, often requiring costly human…
Fantastic Reasoning Behaviors and Where to Find Them: Unsupervised Discovery of the Reasoning Process
Zhenyu Zhang, Shujian Zhang, John Lambert +6
Despite the growing reasoning capabilities of recent large language models (LLMs), their internal mechanisms during the reasoning process remain underexplored. Prior approaches oft…
Eliciting Behaviors in Multi-Turn Conversations
Jing Huang, Shujian Zhang, Lun Wang +3
Identifying specific and often complex behaviors from large language models (LLMs) in conversational settings is crucial for their evaluation. Recent work proposes novel techniques…