Attention via Black-Box Vector Search

Stepan Zharkov, Krish Singal, Ashwin Padaki +1

Sparse attention mechanisms estimate attention over $n$ tokens using a small subset of keys. Many existing approaches use maximum inner product search (MIPS) to retrieve the heavie…