SMART: When is it Actually Worth Expanding a Speculative Tree?

Lifu Wang, Pan Zhou

Tree-based speculative decoding accelerates autoregressive generation by verifying a branching tree of draft tokens in a single target-model forward pass. However, existing methods…