Understanding Verbatim Memorization in LLMs Through Circuit Discovery

Ilya Lasy, Peter Knees, Stefan Woltran

Underlying mechanisms of memorization in LLMs -- the verbatim reproduction of training data -- remain poorly understood. What exact part of the network decides to retrieve a token…