Residual vector quantization for KV cache compression in large language model

Ankur Kumar

KV cache compression methods have mainly relied on scalar quantization techniques to reduce the memory requirements during decoding. In this work, we apply residual vector quantiza…