4 papers · 1 filter
Structuring The Future: Diffusion LLM Speculative Decoding via Calibrated Draft Graphs
Sudhanshu Agrawal, Risheek Garrepalli, Raghavv Goel +3
Diffusion LLMs (dLLMs) have recently emerged as a powerful alternative to autoregressive LLMs (AR-LLMs) with the potential to operate at significantly higher token-generation rates…
VOCABTRIM: Vocabulary Pruning for Efficient Speculative Decoding in LLMs
Raghavv Goel, Sudhanshu Agrawal, Mukul Gagrani +9
In this paper, we introduce a simple training-free technique to improve the performance of drafter-based speculative decoding (SpD) methods that incorporates language modeling head…
KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environments
Junyoung Park, Dalton Jones, Matthew J Morse +3
We demonstrate that geometrically distinctive keys during LLM inference tend to have high attention scores. Based on the phenomenon we propose KeyDiff, a training-free KV cache evi…
CAOTE: KV Cache Selection for LLMs via Attention Output Error-Based Token Eviction
Raghavv Goel, Junyoung Park, Mukul Gagrani +5
While long context support of large language models has extended their abilities, it also incurs challenges in memory and compute which becomes crucial bottlenecks in resource-rest…