Learning Factored Representations in a Deep Mixture of Experts

Abstract

Mixtures of Experts combine the outputs of several "expert" networks, each of which specializes in a different part of the input space. This is achieved by training a "gating" network that maps each input to a distribution over the experts. Such models show promise for building larger networks that are still cheap to compute at test time, and more parallelizable at training time. In this this work, we extend the Mixture of Experts to a stacked model, the Deep Mixture of Experts, with multiple sets of gating and experts. This exponentially increases the number of effective experts by associating each input with a combination of experts at each layer, yet maintains a modest model size. On a randomly translated version of the MNIST dataset, we find that the Deep Mixture of Experts automatically learns to develop location-dependent ("where") experts at the first layer, and class-specific ("what") experts at the second layer. In addition, we see that the different combinations are in use when the model is applied to a dataset of speech monophones. These demonstrate effective use of all expert combinations.

References in corpus (1)

• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation

Cited by in corpus (34)

• Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• Redesigning Multi-Scale Neural Network for Crowd Counting
• A newcomer's guide to deep learning for inverse design in nano-photonics
• Breaking the Softmax Bottleneck: A High-Rank RNN Language Model
• MaskConnect: Connectivity Learning by Gradient Descent
• Dynamic Instance Domain Adaptation
• Hash Layers For Large Sparse Models
• A Tutorial on Deep Latent Variable Models of Natural Language
• Scaling Vision with Sparse Mixture of Experts
• Scalable Transfer Learning with Expert Models
• XNAS: Neural Architecture Search with Expert Advice
• Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts
• Dynamic Deep Neural Networks: Optimizing Accuracy-Efficiency Trade-offs by Selective Execution
• Follow the Prophet: Accurate Online Conversion Rate Prediction in the Face of Delayed Feedback
• Modularity in Deep Learning: A Survey
• Dynamic Multi-Branch Layers for On-Device Neural Machine Translation
• Large-scale Pretraining for Neural Machine Translation with Tens of Billions of Sentence Pairs
• Evaluating Online Continual Learning with CALM
• Rank of Experts: Detection Network Ensemble
• Adaptive Memory Networks
• Fast and Simple Mixture of Softmaxes with BPE and Hybrid-LightRNN for Language Generation
• Multi-modal Experts Network for Autonomous Driving
• Graph Classification by Mixture of Diverse Experts
• Robust Raw Waveform Speech Recognition Using Relevance Weighted Representations
• Selecting Data Adaptive Learner from Multiple Deep Learners using Bayesian Networks
• Switch Spaces: Learning Product Spaces with Sparse Gating
• Question Guided Modular Routing Networks for Visual Question Answering
• Cross-Modal Alignment with Mixture Experts Neural Network for Intral-City Retail Recommendation
• Adversarial Mixture Of Experts with Category Hierarchy Soft Constraint
• Universal Simultaneous Machine Translation with Mixture-of-Experts Wait-k Policy
• Learning in Gated Neural Networks
• Unbiased Gradient Estimation with Balanced Assignments for Mixtures of Experts
• MoEVC: A Mixture-of-experts Voice Conversion System with Sparse Gating Mechanism for Accelerating Online Computation