Training-free Dropout Sampling for Semantic Token Acceptance in Speculative Decoding

Jeongtae Lee, Minjung Jo, Hyunjoon Jeong +5

Speculative decoding accelerates large language model inference by proposing tokens with a lightweight draft model and selectively accepting them using a target model. This work in…