LogQuant: Log-Distributed 2-Bit Quantization of KV Cache with Superior Accuracy Preservation

Han Chen, Zicong Jiang, Zining Zhang +4

We introduce LogQuant, a groundbreaking 2-bit quantization technique for KV Cache in large language model (LLM) inference, delivering substantial memory savings while preserving su…