Fuzzy Speculative Decoding for a Tunable Accuracy-Runtime Tradeoff

Maximilian Holsman, Yukun Huang, Bhuwan Dhingra

Speculative Decoding (SD) enforces strict distributional equivalence to the target model when accepting candidate tokens. While it maintains the target model's generation quality,…