MARS: Unleashing the Power of Speculative Decoding via Margin-Aware Verification

Jingwei Song, Xinyu Wang, Hanbin Wang +6

Speculative Decoding (SD) accelerates autoregressive large language model (LLM) inference by decoupling generation and verification. While recent methods improve draft quality by t…