Delay-Adaptive Speculation Control for Low-Latency Edge-Cloud LLM Inference

Kangkang Sun, Jianhua Li, Xiuzhen Chen +2

Speculative decoding accelerates large language model (LLM) inference by using a lightweight draft model to propose tokens and a larger target model to verify them in parallel. In…