Pipeline Parallelism is All You Need for Optimized Early-Exit Based Self-Speculative Decoding

Ruanjun Li, Ziheng Liu, Yuanming Shi +3

Large language models (LLMs) deliver impressive generation quality, but incur very high inference cost because each output token is generated auto-regressively through all model la…