Don't be so Stief! Learning KV Cache low-rank approximation over the Stiefel manifold

Luca Benfenati, Matteo Risso, Andrea Vannozzi +5

Key-value (KV) caching enables fast autoregressive decoding but at long contexts becomes a dominant bottleneck in High Bandwidth Memory (HBM) capacity and bandwidth. A common mitig…