A mechanistic study of language model introspection

Jiahong Zou, Xiangkun Sun, Lingkai Kong +1

Large language models (LLMs) can sometimes report perturbations to their internal activations---even when the input provides no evidence that an intervention occurred. How do model…