Verification-Aware Training for Speculative Decoding

Geonmo Gu, Byeongho Heo, HeeJae Jun +4

Speculative decoding accelerates large language model inference by using a draft model to generate candidate tokens, which are verified by the target model in a single forward pass…