InAttention: Linear Context Scaling for Transformers

Joseph Eisner

VRAM requirements for transformer models scale quadratically with context length due to the self-attention mechanism. In this paper we modify the decoder-only transformer, replacin…