Mitigating Preference Hacking in Policy Optimization with Pessimism

Dhawal Gupta, Adam Fisch, Christoph Dann +1

This work tackles the problem of overoptimization in reinforcement learning from human feedback (RLHF), a prevalent technique for aligning models with human preferences. RLHF relie…