Spontaneous Reward Hacking in Iterative Self-Refinement

Jane Pan, He He, Samuel R. Bowman +1

Language models are capable of iteratively improving their outputs based on natural language feedback, thus enabling in-context optimization of user preference. In place of human u…