AsymSpec: Efficient Cloud-Edge Speculative Decoding over Asymmetric Networks

Guotao Yang, Hao Chen, Rui Guo +5

Cloud-edge speculative decoding places a lightweight draft model at an edge gateway and a higher-quality target model in the cloud, but inserts communication into every speculative…