Unbiased Gradient Estimation with Balanced Assignments for Mixtures of Experts
arXiv:2109.11817
Abstract
Training large-scale mixture of experts models efficiently on modern hardware requires assigning datapoints in a batch to different experts, each with a limited capacity. Recently proposed assignment procedures lack a probabilistic interpretation and use biased estimators for training. As an alternative, we propose two unbiased estimators based on principled stochastic assignment procedures: one that skips datapoints which exceed expert capacity, and one that samples perfectly balanced assignments using an extension of the Gumbel-Matching distribution [29]. Both estimators are unbiased, as they correct for the used sampling procedure. On a toy experiment, we find the `skip'-estimator is more effective than the balanced sampling one, and both are more robust in solving the task than biased alternatives.
arXiv commentI (Still) Can't Believe It's Not Better Workshop at NeurIPS 2021
References in corpus (17)
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
- Adaptive Computation Time for Recurrent Neural Networks
- Learning Latent Permutations with Gumbel-Sinkhorn Networks
- Conditional Computation in Neural Networks for faster models
- Learning Factored Representations in a Deep Mixture of Experts
- Direct Optimization of Ranking Measures
- Ranking via Sinkhorn Propagation
- BASE Layers: Simplifying Training of Large, Sparse Models
- Hash Layers For Large Sparse Models
- Routing Networks and the Challenges of Modular and Compositional Computation
- Techniques for Learning Binary Stochastic Feedforward Neural Networks
- Stochastic Optimization of Sorting Networks via Continuous Relaxations
- Modular Networks: Learning to Decompose Neural Computation
- Scaling Vision with Sparse Mixture of Experts
- On Sampling from the Gibbs Distribution with Random Maximum A-Posteriori Perturbations
- Approximating the Permanent by Sampling from Adaptive Partitions