KV Cache Compression for Inference Efficiency in LLMs: A Review

Yanyu Liu, Jingying Fu, Sixiang Liu +4

Withtherapid advancement of large language models (LLMs), the context length for inference has been continuously increasing, leading to an exponential growth in the demand for Key-…