Deep Mixture of Experts via Shallow Embedding

Abstract

Larger networks generally have greater representational power at the cost of increased computational complexity. Sparsifying such networks has been an active area of research but has been generally limited to static regularization or dynamic approaches using reinforcement learning. We explore a mixture of experts (MoE) approach to deep dynamic routing, which activates certain experts in the network on a per-example basis. Our novel DeepMoE architecture increases the representational power of standard convolutional networks by adaptively sparsifying and recalibrating channel-wise features in each convolutional layer. We employ a multi-headed sparse gating network to determine the selection and scaling of channels for each input, leveraging exponential combinations of experts within a single convolutional network. Our proposed architecture is evaluated on four benchmark datasets and tasks, and we show that Deep-MoEs are able to achieve higher accuracy with lower computation than standard convolutional networks.

Cited by in corpus (11)

• Dynamic Neural Networks: A Survey
• Zero Time Waste: Recycling Predictions in Early Exit Neural Networks
• Scaling Vision with Sparse Mixture of Experts
• EnsembleNet: End-to-End Optimization of Multi-headed Models
• Generalizable Person Re-identification with Relevance-aware Mixture of Experts
• SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Experts
• Learning to Compose Topic-Aware Mixture of Experts for Zero-Shot Video Captioning
• Breaking the gridlock in Mixture-of-Experts: Consistent and Efficient Algorithms
• Multi-view Depth Estimation using Epipolar Spatio-Temporal Networks
• Learning When to Trust Which Teacher for Weakly Supervised ASR
• Towards a Universal Gating Network for Mixtures of Experts