How to post-train on a surrogate: Envelope sampling mitigates reward hacking

Sanjit Dandapanthula, Shuvom Sadhuka, Samir Khan +3

Large language models (LLMs) are commonly post-trained against LLM judges and other cheap surrogates because the true reward, such as human preference, is too expensive to query at…