HashEvict: A Pre-Attention KV Cache Eviction Strategy using Locality-Sensitive Hashing

Minghui Liu, Tahseen Rabbani, Tony O'Halloran +5

Transformer-based large language models (LLMs) use the key-value (KV) cache to significantly accelerate inference by storing the key and value embeddings of past tokens. However, t…