The Pitfalls of KV Cache Compression

Alex Chen, Renato Geh, Aditya Grover +2

KV cache compression promises increased throughput and efficiency with negligible loss in performance. While the gains in throughput are indisputable and recent literature has inde…