Capacity Matters: a Proof-of-Concept for Transformer Memorization on Real-World Data

Anton Changalidis, Aki Härmä

This paper studies how the model architecture and data configurations influence the empirical memorization capacity of generative transformers. The models are trained using synthet…