FlashOffloading: Low-Latency KV Cache Offloading for Long-Context LLM Serving

Xuan Wang, Zhuanhao Wu, Tingzhu Fang +17

In large language model (LLM) serving systems, the key-value (KV) cache grows linearly with sequence length and concurrency, so long-context inference easily hits the high-bandwidt…