MiniKV: Pushing the Limits of LLM Inference via 2-Bit Layer-Discriminative KV Cache

Akshat Sharma, Hangliang Ding, Jianping Li +2

How to efficiently serve LLMs in practice has become exceptionally challenging due to their prohibitive memory and computation requirements. In this study, we investigate optimizin…