17 papers · 1 filter
Hetu v2: A General and Scalable Deep Learning System with Hierarchical and Heterogeneous Single Program Multiple Data Annotations
Haoyang Li, Fangcheng Fu, Hao Ge +6
The Single-Program Multiple-Data (SPMD) paradigm provides a unified abstraction to annotate various parallel dimensions in distributed deep learning (DL) training. With SPMD, users…
Adaptive Resource Management and Quality Control for Streaming Video Generation
Yifei Xia, Hao Yuan, Suhan Ling +5
Autoregressive diffusion transformers (AR-DiTs) recast video generation from an offline paradigm to a real-time streaming one: the model generates video one chunk at a time, making…
Harnessing Routing Foresight for Micro-step-level MoE load balancing in RL Post-training
Yuming Zhou, Haoyang Li, Sheng Lin +6
Mixture-of-Experts (MoE) and reinforcement learning (RL) post-training now dominate large language model (LLM) development, yet expert load imbalance remains a critical challenge.…
InfiniPipe: Elastic Pipeline Parallelism for Efficient Variable-Length Long-Context LLM Training
Shiju Wang, Yujie Wang, Ao Sun +5
Long context training is crucial for LLM's context extension. Existing schemes, such as sequence parallelism, incur substantial communication overhead. Pipeline parallelism (PP) re…
Efficient Multi-round LLM Inference over Disaggregated Serving
Wenhao He, Youhe Jiang, Penghao Zhao +4
With the rapid evolution of Large Language Models (LLMs), multi-round workflows, such as autonomous agents and iterative retrieval, have become increasingly prevalent. However, thi…
LAER-MoE: Load-Adaptive Expert Re-layout for Efficient Mixture-of-Experts Training
Xinyi Liu, Yujie Wang, Fangcheng Fu +4
Expert parallelism is vital for effectively training Mixture-of-Experts (MoE) models, enabling different devices to host distinct experts, with each device processing different inp…