SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification

Abstract

This paper introduces SpecInfer, a system that accelerates generative large language model (LLM) serving with tree-based speculative inference and verification. The key idea behind SpecInfer is leveraging small speculative models to predict the LLM's outputs; the predictions are organized as a token tree, whose nodes each represent a candidate token sequence. The correctness of all candidate token sequences represented by a token tree is verified against the LLM in parallel using a novel tree-based parallel decoding mechanism. SpecInfer uses an LLM as a token tree verifier instead of an incremental decoder, which significantly reduces the end-to-end latency and computational requirement for serving generative LLMs while provably preserving model quality. Our evaluation shows that SpecInfer outperforms existing LLM serving systems by 1.5-2.8x for distributed LLM inference and by 2.6-3.5x for offloading-based LLM inference, while preserving the same generative performance. SpecInfer is publicly available at https://github.com/flexflow/FlexFlow/

arXiv commentASPLOS'24

References in corpus (11)

• Language Models are Few-Shot Learners
• Ensemble deep learning: A review
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning
• GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
• Beyond Data and Model Parallelism for Deep Neural Networks
• SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot
• FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
• Tree-structured Attention with Hierarchical Accumulation
• Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Masks
• Blockwise Parallel Decoding for Deep Autoregressive Models

Cited by in corpus (7)

• Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems
• Teola: Towards End-to-End Optimization of LLM-based Applications
• On the Cost of Model-Serving Frameworks: An Experimental Evaluation
• Chameleon: Adaptive Caching and Scheduling for Many-Adapter LLM Inference Environments
• AdaSpec: Adaptive Speculative Decoding for Fast, SLO-Aware Large Language Model Serving
• CoDec: Prefix-Shared Decoding Kernel for LLMs
• MemSpec: Memory-Aware Runtime for Adaptive Draft Scheduling in Speculative Decoding on Edge Devices