12 citations · 29 across the 17 of their papers we have counts for
3 papers · 2 filters
Efficient MoE Inference with Fine-Grained Scheduling of Disaggregated Expert Parallelism
Xinglin Pan, Shaohuai Shi, Wenxiang Lin +4
The mixture-of-experts (MoE) architecture scales model size with sublinear computational increase but suffers from memory-intensive inference due to KV caches and sparse expert act…
HierMoE: Accelerating MoE Training with Hierarchical Token Deduplication and Expert Swap
Wenxiang Lin, Xinglin Pan, Lin Zhang +3
The sparsely activated mixture-of-experts (MoE) transformer has become a common architecture for large language models (LLMs) due to its sparsity, which requires fewer computationa…
DreamDDP: Accelerating Data Parallel Distributed LLM Training with Layer-wise Scheduled Partial Synchronization
Zhenheng Tang, Zichen Tang, Junlin Huang +7
The growth of large language models (LLMs) increases challenges of accelerating distributed training across multiple GPUs in different data centers. Moreover, concerns about data p…