Greedy Multi-Path Block Verification for Faster Decoding in Speculative Sampling

Rahul Thomas, Arka Pal, David Hidary

The goal of $L$-step speculative decoding is to accelerate autoregressive decoding of a target model by using a cheaper draft model to generate a candidate path of $L$ tokens. Base…