Ensemble Model Patching: A Parameter-Efficient Variational Bayesian Neural Network

Abstract

Two main obstacles preventing the widespread adoption of variational Bayesian neural networks are the high parameter overhead that makes them infeasible on large networks, and the difficulty of implementation, which can be thought of as "programming overhead." MC dropout [Gal and Ghahramani, 2016] is popular because it sidesteps these obstacles. Nevertheless, dropout is often harmful to model performance when used in networks with batch normalization layers [Li et al., 2018], which are an indispensable part of modern neural networks. We construct a general variational family for ensemble-based Bayesian neural networks that encompasses dropout as a special case. We further present two specific members of this family that work well with batch normalization layers, while retaining the benefits of low parameter and programming overhead, comparable to non-Bayesian training. Our proposed methods improve predictive accuracy and achieve almost perfect calibration on a ResNet-18 trained with ImageNet.

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Weight Uncertainty in Neural Networks
• Understanding deep learning requires rethinking generalization
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Variational Dropout and the Local Reparameterization Trick
• Do ImageNet Classifiers Generalize to ImageNet?
• Averaging Weights Leads to Wider Optima and Better Generalization
• An Empirical Model of Large-Batch Training
• Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors
• Yes, but Did It Work?: Evaluating Variational Inference
• Variational Boosting: Iteratively Refining Posterior Approximations
• Dropout as a Bayesian Approximation: Appendix
• Nonparametric variational inference
• Stochastic Backpropagation through Mixture Density Distributions
• Adversarial Distillation of Bayesian Neural Network Posteriors
• Dynamic Layer Normalization for Adaptive Neural Acoustic Modeling in Speech Recognition

Cited by in corpus (5)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Stacking for Non-mixing Bayesian Computations: The Curse and Blessing of Multimodal Posteriors
• Action and Perception as Divergence Minimization
• Dangers of Bayesian Model Averaging under Covariate Shift
• Improving Classifier Confidence using Lossy Label-Invariant Transformations