Speculative Decoding with a Speculative Vocabulary

Miles Williams, Young D. Kwon, Rui Li +2

Speculative decoding has rapidly emerged as a leading approach for accelerating language model (LM) inference, as it offers substantial speedups while yielding identical outputs. T…