SpecExtend: A Drop-in Enhancement for Speculative Decoding of Long Sequences

Jungyoub Cha, Hyunjong Kim, Sungzoon Cho

Speculative decoding is a widely used technique for accelerating inference in large language models (LLMs), but its performance degrades as input length grows, with significant dro…