Inference-time sparse attention with asymmetric indexing

Pierre-Emmanuel Mazaré, Gergely Szilvasy, Maria Lomeli +4

Self-attention in transformer models is an incremental associative memory that maps key vectors to value vectors. One way to speed up self-attention is to employ GPU-compatible vec…