Clear Preferences Leave Traces: Reference Model-Guided Sampling for Preference Learning

Nirav Diwan, Tolga Ergen, Dongsub Shim +1

Direct Preference Optimization (DPO) has emerged as a de-facto approach for aligning language models with human preferences. Recent work has shown DPO's effectiveness relies on tra…