Accelerating LLM Inference with Staged Speculative Decoding

Benjamin Spector, Chris Re

Recent advances with large language models (LLM) illustrate their diverse capabilities. We propose a novel algorithm, staged speculative decoding, to accelerate LLM inference in sm…