Mitigating KV Cache Competition to Enhance User Experience in LLM Inference

Haiying Shen, Tanmoy Sen, Masahiro Tanaka

In Large Language Model (LLM) serving, the KV-cache (KVC) bottleneck causes high tail Time-to-First-Token (TTFT) and Time-Between-Tokens (TBT), impairing user experience, particula…