Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

Abstract

The capacity of a neural network to absorb information is limited by its number of parameters. Conditional computation, where parts of the network are active on a per-example basis, has been proposed in theory as a way of dramatically increasing model capacity without a proportional increase in computation. In practice, however, there are significant algorithmic and performance challenges. In this work, we address these challenges and finally realize the promise of conditional computation, achieving greater than 1000x improvements in model capacity with only minor losses in computational efficiency on modern GPU clusters. We introduce a Sparsely-Gated Mixture-of-Experts layer (MoE), consisting of up to thousands of feed-forward sub-networks. A trainable gating network determines a sparse combination of these experts to use for each example. We apply the MoE to the tasks of language modeling and machine translation, where model capacity is critical for absorbing the vast quantities of knowledge available in the training corpora. We present model architectures in which a MoE with up to 137 billion parameters is applied convolutionally between stacked LSTM layers. On large language modeling and machine translation benchmarks, these models achieve significantly better results than state-of-the-art at lower computational cost.

Cited by in corpus (34)

• A Neural Dirichlet Process Mixture Model for Task-Free Continual Learning
• A newcomer's guide to deep learning for inverse design in nano-photonics
• Learning from models beyond fine-tuning
• AdaTask: A Task-aware Adaptive Learning Rate Approach to Multi-task Learning
• Routing Networks and the Challenges of Modular and Compositional Computation
• Scaling Vision with Sparse Mixture of Experts
• A visual introduction to Gaussian Belief Propagation
• A Modular Task-oriented Dialogue System Using a Neural Mixture-of-Experts
• A Large Encoder-Decoder Family of Foundation Models For Chemical Language
• FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models
• Finding Fast Transformers: One-Shot Neural Architecture Search by Component Composition
• Deep Networks from the Principle of Rate Reduction
• Modularity in Deep Learning: A Survey
• Fighting Gradients with Gradients: Dynamic Defenses against Adversarial Attacks
• Retrospective and Prospective Mixture-of-Generators for Task-oriented Dialogue Response Generation
• Surprisal-Triggered Conditional Computation with Neural Networks
• Computation on Sparse Neural Networks: an Inspiration for Future Hardware
• Pay Better Attention to Attention: Head Selection in Multilingual and Multi-Domain Sequence Modeling
• One Self-Configurable Model to Solve Many Abstract Visual Reasoning Problems
• Learning Multiplicative Interactions with Bayesian Neural Networks for Visual-Inertial Odometry
• Tiny Models are the Computational Saver for Large Models
• Neurocoder: Learning General-Purpose Computation Using Stored Neural Programs
• Selecting Data Adaptive Learner from Multiple Deep Learners using Bayesian Networks
• Towards More Effective and Economic Sparsely-Activated Model
• Towards Structured Dynamic Sparse Pre-Training of BERT
• DESIRE-ME: Domain-Enhanced Supervised Information REtrieval using Mixture-of-Experts
• Mixture-of-Partitions: Infusing Large Biomedical Knowledge Graphs into BERT
• Double-Wing Mixture of Experts for Streaming Recommendations
• Simple or Complex? Complexity-Controllable Question Generation with Soft Templates and Deep Mixture of Experts Model
• Fast and Flexible Image Blind Denoising via Competition of Experts
• Back to Square One: Superhuman Performance in Chutes and Ladders Through Deep Neural Networks and Tree Search
• Transferring Inter-Class Correlation
• Deep Metric Learning with Locality Sensitive Angular Loss for Self-Correcting Source Separation of Neural Spiking Signals
• ExpertRank: A Multi-level Coarse-grained Expert-based Listwise Ranking Loss