VeriCache: Turning Lossy KV Cache into Lossless LLM Inference

Jiayi Yao, Samuel Shen, Kuntai Du +7

The large size of the KV cache has become a major bottleneck for serving LLMs with increasing context lengths. In response, many KV cache compression methods, such as token droppin…