Preference Tuning as Spectral Update Reorganization

Peiyan Zhang, Haibo Jin, Liying Kang +1

Preference-based post-training is usually understood through endpoint behavior, yet the learned update that produces this behavior remains largely opaque. We study RLHF and related…