Don't Forget! Decomposing the Training Dynamics of Memorization in Language Models

Florian Eichin, Philipp Mondorf, Andrei Mircea +3

Memorization has been proposed as a mechanism to explain how language models fit the tail of their training distributions, but its training dynamics are not understood well. In thi…