2 papers
cs.CL2026
KVCrush: Key value cache size-reduction using similarity in head-behaviour
Gopi Krishna Jha, Sameh Gobriel, Liubov Talamanova +1
Key-value (KV) caching has emerged as a crucial optimization technique for accelerating inference in large language models (LLMs). By allowing the attention operation to scale line…
cs.CL2025
ML-SpecQD: Multi-Level Speculative Decoding with Quantized Drafts
Evangelos Georganas, Dhiraj Kalamkar, Alexander Kozlov +1
Speculative decoding (SD) has emerged as a method to accelerate LLM inference without sacrificing any accuracy over the 16-bit model inference. In a typical SD setup, the idea is t…