Learn from the Past: Fast Sparse Indexing for Large Language Model Decoding

Feiyu Yao, Qian Wang

As large language models (LLMs) continue to support increasingly longer contexts, the memory demand for key-value (KV) caches during decoding grows rapidly, becoming a critical bot…