17 citations · 111 across the 41 of their papers we have counts for
12 papers · 1 filter
Efficient MoE Inference with Fine-Grained Scheduling of Disaggregated Expert Parallelism
Xinglin Pan, Shaohuai Shi, Wenxiang Lin +4
The mixture-of-experts (MoE) architecture scales model size with sublinear computational increase but suffers from memory-intensive inference due to KV caches and sparse expert act…
ElasWave: An Elastic-Native System for Scalable Hybrid-Parallel Training
Xueze Kang, Guangyu Xiang, Yuxin Wang +16
Large-scale LLM pretraining now runs across -- accelerators, making failures routine and elasticity mandatory. We posit that an elastic-native training system must join…
DreamDDP: Accelerating Data Parallel Distributed LLM Training with Layer-wise Scheduled Partial Synchronization
Zhenheng Tang, Zichen Tang, Junlin Huang +7
The growth of large language models (LLMs) increases challenges of accelerating distributed training across multiple GPUs in different data centers. Moreover, concerns about data p…
FusionLLM: A Decentralized LLM Training System on Geo-distributed GPUs with Adaptive Compression
Zhenheng Tang, Xueze Kang, Yiming Yin +11
To alleviate hardware scarcity in training large deep neural networks (DNNs), particularly large language models (LLMs), we present FusionLLM, a decentralized training system desig…
Bandwidth-Aware and Overlap-Weighted Compression for Communication-Efficient Federated Learning
Zichen Tang, Junlin Huang, Rudan Yan +5
Current data compression methods, such as sparsification in Federated Averaging (FedAvg), effectively enhance the communication efficiency of Federated Learning (FL). However, thes…
BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems
Yuxin Wang, Yuhan Chen, Zeyu Li +11
Serving systems for Large Language Models (LLMs) are often optimized to improve quality of service (QoS) and throughput. However, due to the lack of open-source LLM serving workloa…