In-context KV-Cache Eviction for LLMs via Attention-Gate

Zihao Zeng, Bokai Lin, Tianqi Hou +2

The KV-Cache technique has become the standard for the inference of large language models (LLMs). Yet, it is widely criticized that KV-Cache can become a bottleneck of the LLM infe…