A Simple Baseline for Bayesian Uncertainty in Deep Learning

Abstract

We propose SWA-Gaussian (SWAG), a simple, scalable, and general purpose approach for uncertainty representation and calibration in deep learning. Stochastic Weight Averaging (SWA), which computes the first moment of stochastic gradient descent (SGD) iterates with a modified learning rate schedule, has recently been shown to improve generalization in deep learning. With SWAG, we fit a Gaussian using the SWA solution as the first moment and a low rank plus diagonal covariance also derived from the SGD iterates, forming an approximate posterior distribution over neural network weights; we then sample from this Gaussian distribution to perform Bayesian model averaging. We empirically find that SWAG approximates the shape of the true posterior, in accordance with results describing the stationary distribution of SGD iterates. Moreover, we demonstrate that SWAG performs well on a wide variety of tasks, including out of sample detection, calibration, and transfer learning, in comparison to many popular alternatives including MC dropout, KFAC Laplace, SGLD, and temperature scaling.

arXiv commentPublished at NeurIPS 2019

References in corpus (26)

• Auto-Encoding Variational Bayes
• Decoupled Weight Decay Regularization
• Overcoming catastrophic forgetting in neural networks
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Wide Residual Networks
• On Calibration of Modern Neural Networks
• Weight Uncertainty in Neural Networks
• GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Acceleration
• Stochastic Gradient Hamiltonian Monte Carlo
• Regularizing and Optimizing LSTM Language Models
• Variational Dropout and the Local Reparameterization Trick
• Accurate Uncertainties for Deep Learning Using Calibrated Regression
• Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
• Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting
• Essentially No Barriers in Neural Network Energy Landscape
• Evaluating Bayesian Deep Learning Methods for Semantic Segmentation
• Deep Gaussian Processes for Regression using Approximate Expectation Propagation
• Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning
• Gradient Descent Happens in a Tiny Subspace
• The Unusual Effectiveness of Averaging in GAN Training
• The Description Length of Deep Learning Models
• Statistical Inference for Model Parameters in Stochastic Gradient Descent
• Noisy Natural Gradient as Variational Inference
• SWALP : Stochastic Weight Averaging in Low-Precision Training
• Subspace Inference for Bayesian Deep Learning
• Constant Step Size Stochastic Gradient Descent for Probabilistic Modeling

Cited by in corpus (82)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Ensemble Distillation for Robust Model Fusion in Federated Learning
• Modeling the Dynamics of PDE Systems with Physics-Constrained Deep Auto-Regressive Networks
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization
• Practical Deep Learning with Bayesian Principles
• Active learning of deep surrogates for PDEs: Application to metasurface design
• A Survey on Epistemic (Model) Uncertainty in Supervised Learning: Recent Advances and Applications
• Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning
• The Case for Bayesian Deep Learning
• Interpretable Sequence Learning for COVID-19 Forecasting
• Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences
• Confidence Calibration for Convolutional Neural Networks Using Structured Dropout
• Uncertainty in Gradient Boosting via Ensembles
• Tighter risk certificates for neural networks
• The Unreasonable Effectiveness of Deep Evidential Regression
• Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors
• Uncertainty Estimation in Autoregressive Structured Prediction
• Uncertainty Quantification and Deep Ensembles
• Rethinking Parameter Counting in Deep Models: Effective Dimensionality Revisited
• Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty
• On the Validity of Bayesian Neural Networks for Uncertainty Estimation
• Probabilistic Deep Learning to Quantify Uncertainty in Air Quality Forecasting
• Connecting Low-Loss Subspace for Personalized Federated Learning
• Subspace Inference for Bayesian Deep Learning
• Regression Prior Networks
• Bayesian Variational Autoencoders for Unsupervised Out-of-Distribution Detection
• A Weakly Supervised Region-Based Active Learning Method for COVID-19 Segmentation in CT Images
• Bayesian Inference for Large Scale Image Classification
• Bayesian Neural Ordinary Differential Equations
• Robust, Accurate Stochastic Optimization for Variational Inference
• Stochastic Weight Averaging in Parallel: Large-Batch Training that Generalizes Well
• Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantification
• Learning Neural Network Subspaces
• CRUDE: Calibrating Regression Uncertainty Distributions Empirically
• A Quantitative Comparison of Epistemic Uncertainty Maps Applied to Multi-Class Segmentation
• Reliable training and estimation of variance networks
• Estimating Model Uncertainty of Neural Networks in Sparse Information Form
• Towards calibrated and scalable uncertainty representations for neural networks
• Calibration of Model Uncertainty for Dropout Variational Inference
• Bayesian Deep Learning via Subnetwork Inference
• Bayesian active learning for production, a systematic study and a reusable library
• Being Bayesian about Categorical Probability
• A Financial Service Chatbot based on Deep Bidirectional Transformers
• Bayesian Graph Neural Networks for Molecular Property Prediction
• One Versus all for deep Neural Network Incertitude (OVNNI) quantification
• Adaptive Label Smoothing
• Modeling Uncertainty by Learning a Hierarchy of Deep Neural Connections
• Stationary Activations for Uncertainty Calibration in Deep Learning
• Bayesian Inference with Certifiable Adversarial Robustness
• Deep Ensembles from a Bayesian Perspective
• Amortized Conditional Normalized Maximum Likelihood: Reliable Out of Distribution Uncertainty Estimation
• Training on Test Data with Bayesian Adaptation for Covariate Shift
• A comprehensive study on the prediction reliability of graph neural networks for virtual screening
• Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensembling
• Bayesian OOD detection with aleatoric uncertainty and outlier exposure
• A benchmark study on reliable molecular supervised learning via Bayesian learning
• Multi-Loss Sub-Ensembles for Accurate Classification with Uncertainty Estimation
• Distilled Thompson Sampling: Practical and Efficient Thompson Sampling via Imitation Learning
• Deep Curvature Suite
• Improving Uncertainty Calibration of Deep Neural Networks via Truth Discovery and Geometric Optimization
• TRADI: Tracking deep neural network weight distributions for uncertainty estimation
• On the Calibration and Uncertainty of Neural Learning to Rank Models
• Non-Parametric Calibration for Classification
• Meta-Cal: Well-controlled Post-hoc Calibration by Ranking
• Estimating Predictive Uncertainty Under Program Data Distribution Shift
• Empirical Frequentist Coverage of Deep Learning Uncertainty Quantification Procedures
• Are Bayesian neural networks intrinsically good at out-of-distribution detection?
• Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learning
• Predicting the outputs of finite deep neural networks trained with noisy gradients
• Learning a Discriminant Latent Space with Neural Discriminant Analysis
• Scalable Bayesian neural networks by layer-wise input augmentation
• Accurate and Reliable Forecasting using Stochastic Differential Equations
• AutoCP: Automated Pipelines for Accurate Prediction Intervals
• Temporal mixture ensemble models for intraday volume forecasting in cryptocurrency exchange markets
• Probabilistic Safety for Bayesian Neural Networks
• MCU-Net: A framework towards uncertainty representations for decision support system patient referrals in healthcare contexts
• Performance Measurement for Deep Bayesian Neural Network
• Quantifying Uncertainty in Deep Spatiotemporal Forecasting
• Calibrating Structured Output Predictors for Natural Language Processing
• Stochastic Bayesian Neural Networks
• A Novel Unsupervised Post-Processing Calibration Method for DNNS with Robustness to Domain Shift
• Investigating maximum likelihood based training of infinite mixtures for uncertainty quantification