3 papers
cs.CL2025
VOCABTRIM: Vocabulary Pruning for Efficient Speculative Decoding in LLMs
Raghavv Goel, Sudhanshu Agrawal, Mukul Gagrani +9
In this paper, we introduce a simple training-free technique to improve the performance of drafter-based speculative decoding (SpD) methods that incorporates language modeling head…
cs.AI2025
KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environments
Junyoung Park, Dalton Jones, Matthew J Morse +3
We demonstrate that geometrically distinctive keys during LLM inference tend to have high attention scores. Based on the phenomenon we propose KeyDiff, a training-free KV cache evi…
cs.LG2025
CAOTE: KV Cache Selection for LLMs via Attention Output Error-Based Token Eviction
Raghavv Goel, Junyoung Park, Mukul Gagrani +5
While long context support of large language models has extended their abilities, it also incurs challenges in memory and compute which becomes crucial bottlenecks in resource-rest…