10 papers
Charon: A Unified and Fine-Grained Simulator for Large-Scale LLM Training and Inference
Mengtian Yang, Zhekun Zhang, Mingheng Wu +3
Deploying large-scale LLM training and inference with optimal performance is exceptionally challenging due to a complex design space of parallelism strategies, system optimizations…
UniEP: Unified Expert-Parallel MoE MegaKernel for LLM Training
Size Zheng, Xuegui Zheng, Li-wen Chang +1
The exponential growth in Large Language Model (LLM) parameters has transformed model training into an increasingly resource-intensive endeavor. With the stagnation of Moore's Law…
Optimizing Long-context LLM Serving via Fine-grained Sequence Parallelism
Cong Li, Yuzhe Yang, Xuegui Zheng +7
With the advancement of large language models (LLMs), their context windows have rapidly expanded. To meet diverse demands from varying-length requests in online services, existing…
veScale: Consistent and Efficient Tensor Programming with Eager-Mode SPMD
Youjie Li, Cheng Wan, Zhiqi Lin +10
Large Language Models (LLMs) have scaled rapidly in size and complexity, requiring increasingly intricate parallelism for distributed training, such as 3D parallelism. This sophist…
MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism
Ruidong Zhu, Ziheng Jiang, Chao Jin +17
Mixture-of-Experts (MoE) showcases tremendous potential to scale large language models (LLMs) with enhanced performance and reduced computational complexity. However, its sparsely…
Triton-distributed: Programming Overlapping Kernels on Distributed AI Systems with the Triton Compiler
Size Zheng, Wenlei Bao, Qi Hou +19
In this report, we propose Triton-distributed, an extension of existing Triton compiler, to overcome the programming challenges in distributed AI systems. Triton-distributed is the…