4 papers
Beyond Token Eviction: Mixed-Dimension Budget Allocation for Efficient KV Cache Compression
Ruijie Miao, Zhiming Wang, Wang Li +4
Key-value (KV) caching is widely used to accelerate transformer inference, but its memory cost grows linearly with input length, limiting long-context deployment. Existing token ev…
KVReviver: Reversible KV Cache Compression with Sketch-Based Token Reconstruction
Aomufei Yuan, Zhiming Wang, Ruijie Miao +8
As the context length of current large language models (LLMs) rapidly increases, the memory demand for the Key-Value (KV) cache is becoming a bottleneck for LLM deployment and batc…
KeepKV: Achieving Periodic Lossless KV Cache Compression for Efficient LLM Inference
Yuxuan Tian, Zihan Wang, Yebo Peng +6
Efficient inference of large language models (LLMs) is hindered by an ever-growing key-value (KV) cache, making KV cache compression a critical research direction. Traditional meth…
MergeMoE: Efficient Compression of MoE Models via Expert Output Merging
Ruijie Miao, Yilun Yao, Zihan Wang +5
The Mixture-of-Experts (MoE) technique has proven to be a promising solution to efficiently scale the model size, which has been widely applied in recent LLM advancements. However,…