PQ-HSA: Reusing Product-Quantized Scores for Hybrid Sparse-Approximate Attention

Kunming Shao, Jierun Chen, Yanli Wang +4

At each decoding step a language model attends over the key-value (KV) cache of every earlier token, so at long context the attention call is bounded by memory bandwidth. Sparse at…