Themis: Efficient Sparse Model Training Through Fully Sharded Sparse Data Parallelism

Yuhao Qing, Guichao Zhu, Fanxin Li +10

Mixture-of-Experts (MoE) scales large language models cost-effectively, but expert-parallel training suffers severe straggler effects from skewed expert loads. Current systems freq…