Coverage-Driven KV Cache Eviction for Efficient and Improved Inference of LLM

Shuvendu Roy, Mengyao Zhai, Hossein Hajimirsadeghi +1

Large language models (LLMs) excel at complex tasks like question answering and summarization, thanks to their ability to handle long-context inputs. However, deploying LLMs is cos…