Does Self-Evaluation Enable Wireheading in Language Models?

David Demitri Africa, Hans Ethan Ting

Self-evaluation is increasingly central to language model training, underpinning techniques from Constitutional AI to self-refinement. We investigate whether coupling self-evaluati…