Collaborative Speculative Inference for Efficient LLM Inference Serving

Luyao Gao, Jianchun Liu, Hongli Xu +3

Speculative inference is a promising paradigm employing small speculative models (SSMs) as drafters to generate draft tokens, which are subsequently verified in parallel by the tar…