An Interpretable Latency Model for Speculative Decoding in LLM Serving

Linghao Kong, Megan Flynn, Michael Peng +3

Speculative decoding (SD) accelerates large language model (LLM) inference by using a smaller draft model to propose multiple tokens that are verified by a larger target model in p…