Memorization in Attention-only Transformers

Léo Dana, Muni Sreenivas Pydi, Yann Chevaleyre

Recent research has explored the memorization capacity of multi-head attention, but these findings are constrained by unrealistic limitations on the context size. We present a nove…