9 papers
DITRON: Distributed Multi-level Tiling Compiler for Parallel Tensor Programs
Size Zheng, Xuegui Zheng, Hanshi Sun +16
The scaling of large language models (LLMs) is currently bottlenecked by the rigidity of distributed programming. While high-performance libraries like CuBLAS and NCCL provide opti…
UniEP: Unified Expert-Parallel MoE MegaKernel for LLM Training
Size Zheng, Xuegui Zheng, Li-wen Chang +1
The exponential growth in Large Language Model (LLM) parameters has transformed model training into an increasingly resource-intensive endeavor. With the stagnation of Moore's Law…
Optimizing Long-context LLM Serving via Fine-grained Sequence Parallelism
Cong Li, Yuzhe Yang, Xuegui Zheng +7
With the advancement of large language models (LLMs), their context windows have rapidly expanded. To meet diverse demands from varying-length requests in online services, existing…
MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production
Chao Jin, Ziheng Jiang, Zhihao Bai +16
We present MegaScale-MoE, a production system tailored for the efficient training of large-scale mixture-of-experts (MoE) models. MoE emerges as a promising architecture to scale l…
SwiftSpec: Ultra-Low Latency LLM Decoding by Scaling Asynchronous Speculative Decoding
Ziyi Zhang, Ziheng Jiang, Chengquan Jiang +5
Low-latency decoding for large language models (LLMs) is crucial for applications like chatbots and code assistants, yet generating long outputs remains slow in single-query settin…
Triton-distributed: Programming Overlapping Kernels on Distributed AI Systems with the Triton Compiler
Size Zheng, Wenlei Bao, Qi Hou +19
In this report, we propose Triton-distributed, an extension of existing Triton compiler, to overcome the programming challenges in distributed AI systems. Triton-distributed is the…