Speculative Decoding in Decentralized LLM Inference: Turning Communication Latency into Computation Throughput

Jingwei Song, Wanyi Chen, Xinyuan Song +7

Speculative decoding accelerates large language model (LLM) inference by using a lightweight draft model to propose tokens that are later verified by a stronger target model. While…