Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

Abstract

The capacity of a neural network to absorb information is limited by its number of parameters. Conditional computation, where parts of the network are active on a per-example basis, has been proposed in theory as a way of dramatically increasing model capacity without a proportional increase in computation. In practice, however, there are significant algorithmic and performance challenges. In this work, we address these challenges and finally realize the promise of conditional computation, achieving greater than 1000x improvements in model capacity with only minor losses in computational efficiency on modern GPU clusters. We introduce a Sparsely-Gated Mixture-of-Experts layer (MoE), consisting of up to thousands of feed-forward sub-networks. A trainable gating network determines a sparse combination of these experts to use for each example. We apply the MoE to the tasks of language modeling and machine translation, where model capacity is critical for absorbing the vast quantities of knowledge available in the training corpora. We present model architectures in which a MoE with up to 137 billion parameters is applied convolutionally between stacked LSTM layers. On large language modeling and machine translation benchmarks, these models achieve significantly better results than state-of-the-art at lower computational cost.

Cited by in corpus (65)

• Federated Learning for Mobile Keyboard Prediction
• VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts
• Large scale distributed neural network training through online distillation
• AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning
• A Neural Dirichlet Process Mixture Model for Task-Free Continual Learning
• A newcomer's guide to deep learning for inverse design in nano-photonics
• Learning from models beyond fine-tuning
• AdaTask: A Task-aware Adaptive Learning Rate Approach to Multi-task Learning
• ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reduction
• Batch-Shaping for Learning Conditional Channel Gated Networks
• Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety
• Dynamic Neural Networks: A Survey
• Routing Networks and the Challenges of Modular and Compositional Computation
• Scaling Vision with Sparse Mixture of Experts
• Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited
• Wisdom of Committees: An Overlooked Approach To Faster and More Accurate Models
• A visual introduction to Gaussian Belief Propagation
• A Modular Task-oriented Dialogue System Using a Neural Mixture-of-Experts
• A Large Encoder-Decoder Family of Foundation Models For Chemical Language
• Conditional Sum-Product Networks: Imposing Structure on Deep Probabilistic Architectures
• Convergence Rates for Gaussian Mixtures of Experts
• Finding Fast Transformers: One-Shot Neural Architecture Search by Component Composition
• FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models
• On component interactions in two-stage recommender systems
• Lifelong Learning of Compositional Structures
• Deep Convolutional Decision Jungle for Image Classification
• Modularity in Deep Learning: A Survey
• Mixtures of Deep Neural Experts for Automated Speech Scoring
• Improving the Expressiveness of Deep Learning Frameworks with Recursion
• Deep Networks from the Principle of Rate Reduction
• Dynamic Multi-Branch Layers for On-Device Neural Machine Translation
• Fighting Gradients with Gradients: Dynamic Defenses against Adversarial Attacks
• Adaptive Confidence Smoothing for Generalized Zero-Shot Learning
• Multi-Source Cross-Lingual Model Transfer: Learning What to Share
• Network Transplanting
• Pay Better Attention to Attention: Head Selection in Multilingual and Multi-Domain Sequence Modeling
• One Self-Configurable Model to Solve Many Abstract Visual Reasoning Problems
• Computation on Sparse Neural Networks: an Inspiration for Future Hardware
• Surprisal-Triggered Conditional Computation with Neural Networks
• Learning Multiplicative Interactions with Bayesian Neural Networks for Visual-Inertial Odometry
• Retrospective and Prospective Mixture-of-Generators for Task-oriented Dialogue Response Generation
• Tiny Models are the Computational Saver for Large Models
• Modular Universal Reparameterization: Deep Multi-task Learning Across Diverse Domains
• Predefined Sparseness in Recurrent Sequence Models
• Neurocoder: Learning General-Purpose Computation Using Stored Neural Programs
• Efficient Bidirectional Neural Machine Translation
• Selecting Data Adaptive Learner from Multiple Deep Learners using Bayesian Networks
• Towards Structured Dynamic Sparse Pre-Training of BERT
• Anomaly Detection by Recombining Gated Unsupervised Experts
• DESIRE-ME: Domain-Enhanced Supervised Information REtrieval using Mixture-of-Experts
• Towards More Effective and Economic Sparsely-Activated Model
• Double-Wing Mixture of Experts for Streaming Recommendations
• Dynamic Routing Networks
• Mixture-of-Partitions: Infusing Large Biomedical Knowledge Graphs into BERT
• Sparse MoEs meet Efficient Ensembles
• A Neural Tangent Kernel Perspective of Infinite Tree Ensembles
• Fast and Flexible Image Blind Denoising via Competition of Experts
• Interpretable Mixture Density Estimation by use of Differentiable Tree-module
• ExpertRank: A Multi-level Coarse-grained Expert-based Listwise Ranking Loss
• Back to Square One: Superhuman Performance in Chutes and Ladders Through Deep Neural Networks and Tree Search
• Deep Metric Learning with Locality Sensitive Angular Loss for Self-Correcting Source Separation of Neural Spiking Signals
• A method of supervised learning from conflicting data with hidden contexts
• Latent Part-of-Speech Sequences for Neural Machine Translation
• Transferring Inter-Class Correlation
• Simple or Complex? Complexity-Controllable Question Generation with Soft Templates and Deep Mixture of Experts Model