DySpec: Faster Speculative Decoding with Dynamic Token Tree Structure

Yunfan Xiong, Ruoyu Zhang, Yanzeng Li +2

While speculative decoding has recently appeared as a promising direction for accelerating the inference of large language models (LLMs), the speedup and scalability are strongly b…