KVCrush: Key value cache size-reduction using similarity in head-behaviour

Gopi Krishna Jha, Sameh Gobriel, Liubov Talamanova +1

Key-value (KV) caching has emerged as a crucial optimization technique for accelerating inference in large language models (LLMs). By allowing the attention operation to scale line…