Semiparametric Preference Optimization: Your Language Model is Secretly a Single-Index Model

Nathan Kallus

Policy alignment to preference data typically assumes a known link function between observed preferences and latent rewards (e.g., Bradley-Terry model / logistic link). Misspecific…