Understanding Target Probability Reuse in Speculative Decoding

Jungseob Lee, Giyoon Jang, Chailin Nam +6

Speculative decoding uses a target model to verify a tree of draft tokens in parallel. We study whether the resulting next-token distributions can help select the next tree. For un…