83 citations · 244 across the 40 of their papers we have counts for
22 papers · 1 filter
Cobalt: Leveraging Expert Co-activation for Efficient Distributed MoE Training
Junkang Zhou, Xinyi Liu, Fangcheng Fu
Mixture-of-Experts (MoE) has increasingly become a mainstream approach for scaling large language models, as it expands model capacity while keeping computation cost nearly constan…
PipeSwift: Revisiting Pipeline Parallelism for Large-Scale Completion-Oriented Agentic LLM Serving
Shiju Wang, Fei Ren, Fangcheng Fu +4
LLM agents execute long-horizon workflows where each model response determines the progress of subsequent tool interactions and environment transitions. Unlike chatbot serving, whe…
Adaptive Resource Management and Quality Control for Streaming Video Generation
Yifei Xia, Hao Yuan, Suhan Ling +5
Autoregressive diffusion transformers (AR-DiTs) recast video generation from an offline paradigm to a real-time streaming one: the model generates video one chunk at a time, making…
Harnessing Routing Foresight for Micro-step-level MoE load balancing in RL Post-training
Yuming Zhou, Haoyang Li, Sheng Lin +6
Mixture-of-Experts (MoE) and reinforcement learning (RL) post-training now dominate large language model (LLM) development, yet expert load imbalance remains a critical challenge.…
LAER-MoE: Load-Adaptive Expert Re-layout for Efficient Mixture-of-Experts Training
Xinyi Liu, Yujie Wang, Fangcheng Fu +4
Expert parallelism is vital for effectively training Mixture-of-Experts (MoE) models, enabling different devices to host distinct experts, with each device processing different inp…
StaleFlow: Staleness-Aware Data Management for Mitigating Data Skewness in Fully Disaggregated RL Post-Training
Haoyang Li, Sheng Lin, Fangcheng Fu +6
Reinforcement learning (RL) post-training has become pivotal for enhancing the capabilities of modern large models. A recent trend is to develop RL systems with a fully disaggregat…
TridentServe: A Stage-level Serving System for Diffusion Pipelines
Yifei Xia, Fangcheng Fu, Hao Yuan +6
Diffusion pipelines, renowned for their powerful visual generation capabilities, have seen widespread adoption in generative vision tasks (e.g., text-to-image/video). These pipelin…
Concertina: Data-Centric Adaptive Pipeline Parallelism for Efficient Heterogeneous Long-Context LLM Training
Shiju Wang, Yujie Wang, Ao Sun +6
Long context training is crucial for extending LLM context windows. Existing schemes, such as sequence parallelism, incur substantial communication overhead. Pipeline parallelism (…
LobRA: Multi-tenant Fine-tuning over Heterogeneous Data
Sheng Lin, Fangcheng Fu, Haoyang Li +5
With the breakthrough of Transformer-based pre-trained models, the demand for fine-tuning (FT) to adapt the base pre-trained models to downstream applications continues to grow, so…
Galvatron: An Automatic Distributed System for Efficient Foundation Model Training
Xinyi Liu, Yujie Wang, Shenhan Zhu +4
Galvatron is a distributed system for efficiently training large-scale Foundation Models. It overcomes the complexities of selecting optimal parallelism strategies by automatically…
Hetu v2: A General and Scalable Deep Learning System with Hierarchical and Heterogeneous Single Program Multiple Data Annotations
Haoyang Li, Fangcheng Fu, Hao Ge +6
The Single-Program Multiple-Data (SPMD) paradigm provides a unified abstraction to annotate various parallel dimensions in distributed deep learning (DL) training. With SPMD, users…
ByteScale: Efficient Scaling of LLM Training with a 2048K Context Length on More Than 12,000 GPUs
Hao Ge, Junda Feng, Qi Huang +6
Scaling long-context ability is essential for Large Language Models (LLMs). To amortize the memory consumption across multiple devices in long-context training, inter-data partitio…
ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environments
Youhe Jiang, Fangcheng Fu, Xiaozhe Yao +4
Recent developments in large language models (LLMs) have demonstrated their remarkable proficiency in a range of tasks. Compared to in-house homogeneous GPU clusters, deploying LLM…
Demystifying Cost-Efficiency in LLM Serving over Heterogeneous GPUs
Youhe Jiang, Fangcheng Fu, Xiaozhe Yao +6
Recent advancements in Large Language Models (LLMs) have led to increasingly diverse requests, accompanied with varying resource (compute and memory) demands to serve them. However…
Hydraulis: Balancing Large Transformer Model Training via Co-designing Parallel Strategies and Data Assignment
Haoyang Li, Fangcheng Fu, Sheng Lin +8
To optimize large Transformer model training, both efficient parallel computing and advanced data management are indispensable. However, current methods often assume a stable and u…
FlexSP: Accelerating Large Language Model Training via Flexible Sequence Parallelism
Yujie Wang, Shiju Wang, Shenhan Zhu +7
Extending the context length (i.e., the maximum supported sequence length) of LLMs is of paramount significance. To facilitate long context training of LLMs, sequence parallelism h…
LSH-MoE: Communication-efficient MoE Training via Locality-Sensitive Hashing
Xiaonan Nie, Qibin Liu, Fangcheng Fu +6
Larger transformer models always perform better on various tasks but require more costs to scale up the model size. To efficiently enlarge models, the mixture-of-experts (MoE) arch…
Malleus: Straggler-Resilient Hybrid Parallel Training of Large-scale Models via Malleable Data and Model Parallelization
Haoyang Li, Fangcheng Fu, Hao Ge +7
As the scale of models and training data continues to grow, there is an expanding reliance on more GPUs to train large-scale models, which inevitably increases the likelihood of en…
Spindle: Efficient Distributed Training of Multi-Task Large Models via Wavefront Scheduling
Yujie Wang, Shenhan Zhu, Fangcheng Fu +6
Recent foundation models are capable of handling multiple tasks and multiple data modalities with the unified base model structure and several specialized model components. However…
HexiScale: Facilitating Large Language Model Training over Heterogeneous Hardware
Ran Yan, Youhe Jiang, Xiaonan Nie +3
Training large language models (LLMs) is a computationally intensive task, which is typically conducted in data centers with homogeneous high-performance GPUs. In this paper, we ex…
OSDP: Optimal Sharded Data Parallel for Distributed Deep Learning
Youhe Jiang, Fangcheng Fu, Xupeng Miao +2
Large-scale deep learning models contribute to significant performance improvements on varieties of downstream tasks. Current data and model parallelism approaches utilize model re…
K-Core Decomposition on Super Large Graphs with Limited Resources
Shicheng Gao, Jie Xu, Xiaosen Li +5
K-core decomposition is a commonly used metric to analyze graph structure or study the relative importance of nodes in complex graphs. Recent years have seen rapid growth in the sc…