8 papers
InstantInfer: Enabling Fast LLM Cold Start with Communicating Finite Automata
Yitao Yuan, Yongchao He, Shaoke Fang +1
Cold starts in large language model (LLM) inference services significantly affect user experience, yet they remain inefficient due to sequential initialization and a massive number…
FluxMoE: Decoupling Expert Residency for High-Performance MoE Serving
Qingxiu Liu, Yongchao He, Runhan Jiang +4
Mixture-of-Experts (MoE) models have become mainstream for scaling language models to hundreds of billions of expert parameters. Despite sparse expert activation, existing inferenc…
A Unified Sparse Attention via Multi-Granularity Compression
Siran Liu, Zane Cao, Yongchao He
Efficient long-context understanding and reasoning are increasingly vital for large language model (LLM) applications such as multi-turn dialogue and program analysis. However, the…
CascadeInfer: Length-Aware Scheduling of LLM Serving with Low Latency and Load Balancing
Yitao Yuan, Chenqi Zhao, Bohan Zhao +3
Efficiently harnessing GPU compute is critical to improving user experience and reducing operational costs in large language model (LLM) services. However, current inference engine…
SIMPLE: Disaggregating Sampling from GPU Inference into a Decision Plane for Faster Distributed LLM Serving
Bohan Zhao, Zane Cao, Yongchao He
As large language models (LLMs) scale out with tensor parallelism (TP) and pipeline parallelism (PP) and production stacks have aggressively optimized the data plane (attention/GEM…
MegatronApp: Efficient and Comprehensive Management on Distributed LLM Training
Bohan Zhao, Guang Yang, Shuo Chen +4
The rapid escalation in the parameter count of large language models (LLMs) has transformed model training from a single-node endeavor into a highly intricate, cross-node activity.…
SiPipe: Bridging the CPU-GPU Utilization Gap for Efficient Pipeline-Parallel LLM Inference
Yongchao He, Bohan Zhao, Zheng Cao
As inference workloads for large language models (LLMs) scale to meet growing user demand, pipeline parallelism (PP) has become a widely adopted strategy for multi-GPU deployment,…
HeteroSpec: Leveraging Contextual Heterogeneity for Efficient Speculative Decoding
Siran Liu, Yang Ye, Qianchao Zhu +2
Autoregressive decoding inherently limits the inference throughput of Large Language Model (LLM) due to its sequential dependency. Speculative decoding mitigates this by verifying…