LoLA: Low-Rank Linear Attention With Sparse Caching

Luke McDermott, Robert W. Heath, Rahul Parhi

The per-token cost of transformer inference scales with context length, preventing its application to lifelong in-context learning. Linear attention is an efficient alternative tha…