Soft Preference Optimization: Aligning Language Models to Expert Distributions

Arsalan Sharifnassab, Saber Salehkaleybar, Sina Ghiassian +2

We propose Soft Preference Optimization (SPO), a method for aligning generative models, such as Large Language Models (LLMs), with human preferences, without the need for a reward…