KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference

Jian Lin, Jiazhi Mi, Zicong Hong +5

Supporting long-context LLMs is challenging due to the substantial memory demands of the key-value (KV) cache. Existing offloading systems store the full cache in host memory and s…