Osprey: Target-agnostic Pre-training Makes Stronger Drafters in Speculative Decoding

Fengxiang Bie, Yuqing Jian, Yifan Yu +7

Speculative decoding is critical for accelerating LLM inference. However, the speedup is fragile: drafters are typically trained against a narrow distribution for a single target m…