SPIN: Accelerating Large Language Model Inference with Heterogeneous Speculative Models

Fahao Chen, Peng Li, Tom H. Luan +2

Speculative decoding has been shown as an effective way to accelerate Large Language Model (LLM) inference by using a Small Speculative Model (SSM) to generate candidate tokens in…