From Positionwise Confidence to Prefix Scheduling: Verifier Skipping in Speculative Decoding

Haoxuan Luo, Jameson Sandler, Ferdinando Fioretto

Speculative decoding is a leading technique to reduce the cost of autoregressive generation by using a small drafter to propose several tokens, which are then verified in parallel…