RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference

Wenhao Li, Jinhao Dong, Hailin Zhang +3

Long-context Large Language Model inference is severely bottlenecked by the massive Key-Value (KV) cache, yet existing sparse attention methods often suffer from static fixed-budge…