Preference Instability in Reward Models: Detection and Mitigation via Sparse Autoencoders

Shunchang Liu, Xin Chen, Belen Martin Urcelay +1

Preference learning in large language models relies on reward models as proxies for human judgment. However, these models frequently exhibit preference instability, producing contr…