Adaptive KV Cache Reuse for Fast Long-Context LLM Serving

Fei li, Song Liu, Yan Liu +4

In long-context Large Language Model (LLM) inference, the Time-To-First-Token (TTFT) latency incurred by the prefill stage has become the foremost bottleneck limiting interactive p…