149 citations · 156 across the 7 of their papers we have counts for
Showing 2024Show all
3 papers · 1 filter
cs.DC2024★ 5 cited
MoC-System: Efficient Fault Tolerance for Sparse Mixture-of-Experts Model Training
Weilin Cai, Le Qin, Jiayi Huang
As large language models continue to scale up, distributed training systems have expanded beyond 10k nodes, intensifying the importance of fault tolerance. Checkpoint has emerged a…
cs.LG2024★ 149 cited
A Survey on Mixture of Experts in Large Language Models
Weilin Cai, Juyong Jiang, Fan Wang +3
Large language models (LLMs) have garnered unprecedented advancements across diverse fields, ranging from natural language processing to computer vision and beyond. The prowess of…
cs.LG2024★ 2 cited
Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Experts
Weilin Cai, Juyong Jiang, Le Qin +3
Expert parallelism has emerged as a key strategy for distributing the computational workload of sparsely-gated mixture-of-experts (MoE) models across multiple devices, enabling the…