Intelligently Weighting Multiple Reference Models for Direct Preference Optimization of LLMs

Skyler Wu, Aymen Echarghaoui

Fine-tuning is integral for aligning large language models (LLMs) with human preferences. Multiple-Reference Preference Optimization (MRPO) builds on Direct Preference Optimization…