CentroidKV: Efficient Long-Context LLM Inference via KV Cache Clustering

Jie Hu, Shengnan Wang, Yutong He +8

Large language models (LLMs) with extended context windows have become increasingly prevalent for tackling complex tasks. However, the substantial Key-Value (KV) cache required for…