Vegas: Self-Speculative Decoding with Verification-Guided Sparse Attention

Yikang Yue, Yuqi Xue, Jian Huang

Long-context large language model (LLM) inference has become the norm for today's AI applications. However, it is severely bottlenecked by the increasing memory demands of its KV c…