FlowSpec: Continuous Pipelined Speculative Decoding for Efficient Distributed LLM Inference

Xing Liu, Lizhuo Luo, Ming Tang +2

Distributed inference serves as a promising approach to enabling the inference of large language models (LLMs) at the network edge. It distributes the inference process to multiple…