Structured Dropout Variational Inference for Bayesian Neural Networks

Abstract

Approximate inference in Bayesian deep networks exhibits a dilemma of how to yield high fidelity posterior approximations while maintaining computational efficiency and scalability. We tackle this challenge by introducing a novel variational structured approximation inspired by the Bayesian interpretation of Dropout regularization. Concretely, we focus on the inflexibility of the factorized structure in Dropout posterior and then propose an improved method called Variational Structured Dropout (VSD). VSD employs an orthogonal transformation to learn a structured representation on the variational Gaussian noise with plausible complexity, and consequently induces statistical dependencies in the approximate posterior. Theoretically, VSD successfully addresses the pathologies of previous Variational Dropout methods and thus offers a standard Bayesian justification. We further show that VSD induces an adaptive regularization term with several desirable properties which contribute to better generalization. Finally, we conduct extensive experiments on standard benchmarks to demonstrate the effectiveness of VSD over state-of-the-art variational methods on predictive accuracy, uncertainty estimation, and out-of-distribution detection.

arXiv comment45 pages, 9 figures

References in corpus (36)

• Adam: A Method for Stochastic Optimization
• Auto-Encoding Variational Bayes
• Improving neural networks by preventing co-adaptation of feature detectors
• On Calibration of Modern Neural Networks
• Variational Inference with Normalizing Flows
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
• Bayesian Convolutional Neural Networks with Bernoulli Approximate Variational Inference
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• The Marginal Value of Adaptive Gradient Methods in Machine Learning
• Variational Dropout and the Local Reparameterization Trick
• Improving Generalization Performance by Switching from Adam to SGD
• Bayesian Compression for Deep Learning
• Variational Dropout Sparsifies Deep Neural Networks
• A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
• Dropout Training as Adaptive Regularization
• Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization
• Bayesian Uncertainty Estimation for Batch Normalized Deep Networks
• Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
• Practical Deep Learning with Bayesian Principles
• Sylvester Normalizing Flows for Variational Inference
• Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriors
• A PAC-Bayesian Tutorial with A Dropout Bound
• A Bayesian encourages dropout
• On the Implicit Bias of Dropout
• On the Importance of Strong Baselines in Bayesian Deep Learning
• SLANG: Fast Structured Covariance Approximations for Bayesian Deep Learning with Natural Gradient
• Dropout as a Structured Shrinkage Prior
• Improving Variational Auto-Encoders using convex combination linear Inverse Autoregressive Flow
• On Convergence and Generalization of Dropout Training
• On Dropout and Nuclear Norm Regularization
• Machine Learning's Dropout Training is Distributionally Robust Optimal
• A Unified Approach to Adaptive Regularization in Online and Stochastic Optimization
• Radial and Directional Posteriors for Bayesian Neural Networks
• Variational Dropout via Empirical Bayes
• Walsh-Hadamard Variational Inference for Bayesian Deep Learning