Flatter Tokens are More Valuable for Speculative Draft Model Training

Jiaming Fan, Daming Cao, Xiangzhong Luo +3

Speculative Decoding (SD) is a key technique for accelerating Large Language Model (LLM) inference, but it typically requires training a draft model on a large dataset. We approach…