REFA: Reference Free Alignment for multi-preference optimization

Taneesh Gupta, Rahul Madhavan, Xuchao Zhang +2

To mitigate reward hacking from response verbosity, modern preference optimization methods are increasingly adopting length normalization (e.g., SimPO, ORPO, LN-DPO). While effecti…