Fast Collaborative Inference via Distributed Speculative Decoding

Ce Zheng, Ke Zhang, Chen Sun +3

Speculative decoding accelerates large language model (LLM) inference by allowing a small draft model to predict multiple future tokens for verification by a larger target model. I…