Distance-KV: Exploiting Relative Distance for Efficient Long-Context Inference

Xianpeng Shang, Canbin Huang, Jiang Li +4

The memory usage and decoding latency of LLM inference grow rapidly with context length. To reduce these costs, key-value (KV) cache compression methods selectively retain cached s…