Bayesian Uncertainty Estimation for Batch Normalized Deep Networks

Abstract

We show that training a deep network using batch normalization is equivalent to approximate inference in Bayesian models. We further demonstrate that this finding allows us to make meaningful estimates of the model uncertainty using conventional architectures, without modifications to the network or the training procedure. Our approach is thoroughly validated by measuring the quality of uncertainty in a series of empirical experiments on different tasks. It outperforms baselines with strong statistical significance, and displays competitive performance with recent Bayesian approaches.

ICML 2018

References in corpus (5)

• Auto-Encoding Variational Bayes
• Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
• Fully Convolutional Attention Networks for Fine-Grained Recognition
• Deep Gaussian Processes for Regression using Approximate Expectation Propagation
• Dropout Inference in Bayesian Neural Networks with Alpha-divergences

Cited by in corpus (34)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges
• Recent Advances and Applications of Deep Learning Methods in Materials Science
• Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks
• Uncertainty Quantification in Machine Learning for Engineering Design and Health Prognostics: A Tutorial
• L1-Norm Batch Normalization for Efficient Training of Deep Neural Networks
• A Survey on Epistemic (Model) Uncertainty in Supervised Learning: Recent Advances and Applications
• Mapping oil palm density at country scale: An active learning approach
• Large-Scale Visual Active Learning with Deep Probabilistic Ensembles
• Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty
• Inhibited Softmax for Uncertainty Estimation in Neural Networks
• Bayesian Inference for Large Scale Image Classification
• A Quantitative Comparison of Epistemic Uncertainty Maps Applied to Multi-Class Segmentation
• Hyperparameter-Free Out-of-Distribution Detection Using Softmax of Scaled Cosine Similarity
• Learning Uncertainty For Safety-Oriented Semantic Segmentation In Autonomous Driving
• DS-UI: Dual-Supervised Mixture of Gaussian Mixture Models for Uncertainty Inference
• Quantifying sources of uncertainty in drug discovery predictions with probabilistic models
• SSN: Learning Sparse Switchable Normalization via SparsestMax
• Attending to Discriminative Certainty for Domain Adaptation
• Proxy-Normalizing Activations to Match Batch Normalization while Removing Batch Dependence
• Exploring Uncertainty in Deep Learning for Construction of Prediction Intervals
• Noisy Batch Active Learning with Deterministic Annealing
• Gibbs Sampling the Posterior of Neural Networks
• The Monte Carlo Transformer: a stochastic self-attention model for sequence prediction
• TRADI: Tracking deep neural network weight distributions for uncertainty estimation
• Effect of latent space distribution on the segmentation of images with multiple annotations
• Structured Dropout Variational Inference for Bayesian Neural Networks
• Uncertainty-Aware COVID-19 Detection from Imbalanced Sound Data
• Controllable Orthogonalization in Training DNNs
• Group Whitening: Balancing Learning Efficiency and Representational Capacity
• Diagnostic Uncertainty Calibration: Towards Reliable Machine Predictions in Medical Domain
• An Investigation into the Stochasticity of Batch Whitening
• Training BatchNorm Only in Neural Architecture Search and Beyond
• Uncertainty based Class Activation Maps for Visual Question Answering