End-to-End Context Compression at Scale

Ang Li, Sean McLeish, Haozhe Chen +12

Long-context language model inference is bottlenecked by memory, as the KV cache grows with context length. Recent techniques to compress the KV cache fall short: they either degra…