KVComp: A High-Performance, LLM-Aware, Lossy Compression Framework for KV Cache

Bo Jiang, Taolue Yang, Youyuan Liu +3

Transformer-based large language models (LLMs) demonstrate impressive potential in various practical applications. However, long context inference poses a significant challenge due…