Open Problems in Constitutional Preference Reconstruction

Eleanor Clifford, Michael Amir, Arduin Findeis +2

Pairwise preference data is widely used for training and evaluating language models (e.g., RLHF), but each datapoint records a \emph{choice}, not the rationale behind it. Methods s…