SPIRe: Boosting LLM Inference Throughput with Speculative Decoding

Sanjit Neelam, Daniel Heinlein, Vaclav Cvicek +2

Speculative decoding (SD) has been shown to reduce the latency of autoregressive decoding (AD) by 2-3x for small batch sizes. However, increasing throughput and therefore reducing…