MosaicKV: Serving Long-Context LLM with Dynamic Two-D KV Cache Compression

Sheng Qiang, Ruiwei Chen, Yinpeng Wu +5

Long-context LLM services now sustain prompts with hundreds of thousands to millions of tokens, making the key-value (KV) cache a first-order serving cost. Because the cache grows…