SpecPipe: Accelerating Pipeline Parallelism-based LLM Inference with Speculative Decoding

Haofei Yin, Mengbai Xiao, Tinghong Li +3

The demand for large language model inference is rapidly increasing. Pipeline parallelism offers a cost-effective deployment strategy for distributed inference but suffers from hig…