OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structure

Jikai Wang, Yi Su, Juntao Li +5

Autoregressive language models demonstrate excellent performance in various scenarios. However, the inference efficiency is limited by its one-step-one-word generation mode, which…