Speculative Verification: Exploiting Information Gain to Refine Speculative Decoding

Sungkyun Kim, Jaemin Kim, Dogyung Yoon +3

LLMs have low GPU efficiency and high latency due to autoregressive decoding. Speculative decoding (SD) mitigates this using a small draft model to speculatively generate multiple…