SharpDraft: Accelerating Long-Context Speculative Decoding with Cardinality-Aware Query Scaling

Jeonghoon Park, Seongwoon Jo, Jongwon Lee +1

Long-form reasoning makes inference expensive, and speculative decoding mitigates this cost by verifying multiple draft tokens in parallel. Its speedup, however, can fade as contex…