Mixture of Chapters: Scaling Learnt Memory in Transformers

Tasmay Pankaj Tibrewal, Pritish Saha, Ankit Meda +2

Transformers lack an explicit architectural mechanism for storing and organizing knowledge acquired during training. We introduce learnable sparse memory banks: a set of latent tok…