Test-Time Speculation

Avinash Kumar, Sujay Sanghavi, Poulami Das

Speculative decoding accelerates LLM inference by using a fast draft model to generate tokens and a more accurate target model to verify them. Its performance depends on the $\text…