LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy

Rongzhi Zhang, Kuang Wang, Liyuan Liu +4

The Key-Value (KV) cache is a crucial component in serving transformer-based autoregressive large language models (LLMs), enabling faster inference by storing previously computed K…