Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity

Da Ma, Lu Chen, Situo Zhang +8

The rapid expansion of context window sizes in Large Language Models~(LLMs) has enabled them to tackle increasingly complex tasks involving lengthy documents. However, this progres…