Twig: Parallel and Memory-Efficient Tree Speculative Decoding for Hybrid-Attention LLMs

Li Wang, Yi Su, Xiabao Wu +9

Emerging LLMs increasingly interleave full attention with recurrent linear attention to reduce long-context serving costs. Tree speculative decoding could further accelerate these…