Extended Mind Transformers

Phoebe Klett, Thomas Ahle

Pre-trained language models demonstrate general intelligence and common sense, but long inputs quickly become a bottleneck for memorizing information at inference time. We resurfac…