Sub-Token Routing for KV Cache Compression

Wei Jiang, Wei Wang

Transformer inference often requires a large KV cache, especially for long-context language modeling and multimodal generation. Existing compression methods usually reduce cache co…