Themis: Efficient Sparse Model Training Through Fully Sharded Sparse Data Parallelism

Abstract

Mixture-of-Experts (MoE) scales large language models cost-effectively, but expert-parallel training suffers severe straggler effects from skewed expert loads. Current systems frequently rearrange expert placement to mitigate stragglers, inflating memory footprint and migration overhead, potentially negating the benefits of load balancing. We present Fully Sharded Sparse Data Parallelism (FSSDP), a sparse-native MoE training approach that enables in-situ load balancing on every training iteration, overlapping the balancing with computation and eliminating explicit expert rearrangement together with its migration traffic and memory reserves. FSSDP keeps MoE layers sharded and sparsely materializes an ephemeral, load-balancing placement each iteration, with re-materialization to reuse available memory across layers. FSSDP is complemented by heterogeneous sharding to shift memory imbalance from the device level to the layer level, maintaining uniform memory budgets while enabling per-layer placement optimization. We realize FSSDP in Themis with co-designed topology-aware placement algorithms. Across 2 clusters and diverse workloads, Themis achieves 1.26-2.42x speedup over state-of-the-art expert-rearrangement systems.