M6-T: Exploring Sparse Expert Models and Beyond

Abstract

Mixture-of-Experts (MoE) models can achieve promising results with outrageous large amount of parameters but constant computation cost, and thus it has become a trend in model scaling. Still it is a mystery how MoE layers bring quality gains by leveraging the parameters with sparse activation. In this work, we investigate several key factors in sparse expert models. We observe that load imbalance may not be a significant problem affecting model quality, contrary to the perspectives of recent studies, while the number of sparsely activated experts $k$ and expert capacity $C$ in top-$k$ routing can significantly make a difference in this context. Furthermore, we take a step forward to propose a simple method called expert prototyping that splits experts into different prototypes and applies $k$ top-$1$ routing. This strategy improves the model quality but maintains constant computational costs, and our further exploration on extremely large-scale models reflects that it is more effective in training larger models. We push the model scale to over $1$ trillion parameters and implement it on solely $480$ NVIDIA V100-32GB GPUs, in comparison with the recent SOTAs on $2048$ TPU cores. The proposed giant model achieves substantial speedup in convergence over the same-size baseline.

16 pages, 8 figures

References in corpus (11)

• Decoupled Weight Decay Regularization
• Scaling Laws for Neural Language Models
• Zero-Shot Text-to-Image Generation
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph
• ZeRO-Offload: Democratizing Billion-Scale Model Training
• ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning
• M6: A Chinese Multimodal Pretrainer
• FastMoE: A Fast Mixture-of-Expert Training System

Cited by in corpus (1)

• M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining