ContiguousKV: Accelerating LLM Prefill with Granularity-Aligned KV Cache Management

Jing Zou, Shangyu Wu, Hancong Duan +2

Efficiently serving Large Language Models (LLMs) with persistent Prefix Key-Value (KV) Cache is critical for applications like conversational search and multi-turn dialogue. Servin…