Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles

Abstract

Deep neural networks (NNs) are powerful black box predictors that have recently achieved impressive performance on a wide spectrum of tasks. Quantifying predictive uncertainty in NNs is a challenging and yet unsolved problem. Bayesian NNs, which learn a distribution over weights, are currently the state-of-the-art for estimating predictive uncertainty; however these require significant modifications to the training procedure and are computationally expensive compared to standard (non-Bayesian) NNs. We propose an alternative to Bayesian NNs that is simple to implement, readily parallelizable, requires very little hyperparameter tuning, and yields high quality predictive uncertainty estimates. Through a series of experiments on classification and regression benchmarks, we demonstrate that our method produces well-calibrated uncertainty estimates which are as good or better than approximate Bayesian NNs. To assess robustness to dataset shift, we evaluate the predictive uncertainty on test examples from known and unknown distributions, and show that our method is able to express higher uncertainty on out-of-distribution examples. We demonstrate the scalability of our method by evaluating predictive uncertainty estimates on ImageNet.

NIPS 2017

References in corpus (6)

• Distilling the Knowledge in a Neural Network
• Ensemble Adversarial Training: Attacks and Defenses
• On Calibration of Modern Neural Networks
• Weight Uncertainty in Neural Networks
• Snapshot Ensembles: Train 1, get M for free
• A Bayesian encourages dropout

Cited by in corpus (115)

• On Calibration of Modern Neural Networks
• Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods
• Recent Advances and Applications of Deep Learning Methods in Materials Science
• Bayesian Deep Convolutional Encoder-Decoder Networks for Surrogate Modeling and Uncertainty Quantification
• Accurate Uncertainties for Deep Learning Using Calibrated Regression
• Multiplicative Normalizing Flows for Variational Bayesian Neural Networks
• Text2Motion: From Natural Language Instructions to Feasible Plans
• An Uncertainty-aware Transfer Learning-based Framework for Covid-19 Diagnosis
• Galaxy Zoo: Probabilistic Morphology through Bayesian CNNs and Active Learning
• The Fishyscapes Benchmark: Measuring Blind Spots in Semantic Segmentation
• Physics guided machine learning using simplified theories
• AdaFuse: Adaptive Multiview Fusion for Accurate Human Pose Estimation in the Wild
• Deep Confidence: A Computationally Efficient Framework for Calculating Reliable Errors for Deep Neural Networks
• The Conditional Entropy Bottleneck
• Implicit Full Waveform Inversion with Deep Neural Representation
• The Pitfalls of Simplicity Bias in Neural Networks
• Deeply Uncertain: Comparing Methods of Uncertainty Quantification in Deep Learning Algorithms
• Fast Uncertainty Estimates in Deep Learning Interatomic Potentials
• i-PI 3.0: a flexible and efficient framework for advanced atomistic simulations
• Uncertainty in the Variational Information Bottleneck
• QU-BraTS: MICCAI BraTS 2020 Challenge on Quantifying Uncertainty in Brain Tumor Segmentation - Analysis of Ranking Scores and Benchmarking Results
• Model fusion with physics-guided machine learning
• Deep Ensembles vs. Committees for Uncertainty Estimation in Neural-Network Force Fields: Comparison and Application to Active Learning
• Automated quantification of myocardial tissue characteristics from native T1 mapping using neural networks with Bayesian inference for uncertainty-based quality-control
• Artificial Neural Networks Modelling of Wall Pressure Spectra Beneath Turbulent Boundary Layers
• DeepTriangle: A Deep Learning Approach to Loss Reserving
• Conformal Prediction Under Feedback Covariate Shift for Biomolecular Design
• Reliability and Interpretability in Science and Deep Learning
• Artificial intelligence approaches for materials-by-design of energetic materials: state-of-the-art, challenges, and future directions
• A Smooth Representation of Belief over SO(3) for Deep Rotation Learning with Uncertainty
• Uncertainty Averse Pushing with Model Predictive Path Integral Control
• The Unreasonable Effectiveness of Deep Evidential Regression
• Uncertainty-Guided Efficient Interactive Refinement of Fetal Brain Segmentation from Stacks of MRI Slices
• Monitoring and explainability of models in production
• Learning Robotic Navigation from Experience: Principles, Methods, and Recent Results
• Outside the Box: Abstraction-Based Monitoring of Neural Networks
• Generative machine learning methods for multivariate ensemble post-processing
• Out-of-Distribution Detection using Multiple Semantic Label Representations
• Probabilistic Deep Learning to Quantify Uncertainty in Air Quality Forecasting
• Coarse to Fine Multi-Resolution Temporal Convolutional Network
• A Comparison of Uncertainty Estimation Approaches in Deep Learning Components for Autonomous Vehicle Applications
• Physics-informed Bayesian inference of external potentials in classical density-functional theory
• Don't Just Blame Over-parametrization for Over-confidence: Theoretical Analysis of Calibration in Binary Classification
• Anytime Inference with Distilled Hierarchical Neural Ensembles
• Confidence-Aware Learning for Camouflaged Object Detection
• High carbon stock mapping at large scale with optical satellite imagery and spaceborne LIDAR
• Adaptive Semantic Segmentation with a Strategic Curriculum of Proxy Labels
• A Quantitative Comparison of Epistemic Uncertainty Maps Applied to Multi-Class Segmentation
• A Gaussian Process perspective on Convolutional Neural Networks
• RDeepSense: Reliable Deep Mobile Computing Models with Uncertainty Estimations
• SoK: Machine Learning Governance
• A Unified Benchmark for the Unknown Detection Capability of Deep Neural Networks
• Uncertainty Quantification in CNN Through the Bootstrap of Convex Neural Networks
• Detecting OODs as datapoints with High Uncertainty
• When and How Mixup Improves Calibration
• Generalized Negative Correlation Learning for Deep Ensembling
• Confidence Aware Neural Networks for Skin Cancer Detection
• Uncertainty-Encoded Multi-Modal Fusion for Robust Object Detection in Autonomous Driving
• Influence of uncertainty estimation techniques on false-positive reduction in liver lesion detection
• Deep transformation models for functional outcome prediction after acute ischemic stroke
• Scalable marginalization of correlated latent variables with applications to learning particle interaction kernels
• Enhanced sampling of robust molecular datasets with uncertainty-based collective variables
• Synergizing between Self-Training and Adversarial Learning for Domain Adaptive Object Detection
• Powerpropagation: A sparsity inducing weight reparameterisation
• Ensemble Neural Networks for Remaining Useful Life (RUL) Prediction
• Protecting Classifiers From Attacks
• Fast and reliable uncertainty quantification with neural network ensembles for industrial image classification
• Ranking over Regression for Bayesian Optimization and Molecule Selection
• Strength in Numbers: Trading-off Robustness and Computation via Adversarially-Trained Ensembles
• Low Emission Building Control with Zero-Shot Reinforcement Learning
• Learning Graph Neural Networks for Multivariate Time Series Anomaly Detection
• Pixel-wise Energy-biased Abstention Learning for Anomaly Segmentation on Complex Urban Driving Scenes
• SLURP: Side Learning Uncertainty for Regression Problems
• Learning to Map for Active Semantic Goal Navigation
• On Calibration of Mixup Training for Deep Neural Networks
• TreeFlow: Going beyond Tree-based Gaussian Probabilistic Regression
• Gradual Domain Adaptation via Self-Training of Auxiliary Models
• Effect of latent space distribution on the segmentation of images with multiple annotations
• Exploring Covariate and Concept Shift for Detection and Calibration of Out-of-Distribution Data
• Channel-Driven Monte Carlo Sampling for Bayesian Distributed Learning in Wireless Data Centers
• Data-driven geophysical forecasting: Simple, low-cost, and accurate baselines with kernel methods
• Ex uno plures: Splitting One Model into an Ensemble of Subnetworks
• Using Artificial Populations to Study Psychological Phenomena in Neural Models
• Deep-learning-driven Reliable Single-pixel Imaging with Uncertainty Approximation
• Estimating Predictive Uncertainty Under Program Data Distribution Shift
• Structured Dropout Variational Inference for Bayesian Neural Networks
• Meta-Cal: Well-controlled Post-hoc Calibration by Ranking
• Empirical Frequentist Coverage of Deep Learning Uncertainty Quantification Procedures
• Bayesian Neural Networks with Soft Evidence
• Iterative Distillation for Better Uncertainty Estimates in Multitask Emotion Recognition
• Offline Preference-Based Apprenticeship Learning
• The Compact Support Neural Network
• Learning Off-Policy with Online Planning
• On Efficient Uncertainty Estimation for Resource-Constrained Mobile Applications
• Robust Partial-Label Learning by Leveraging Class Activation Values
• Towards Robust Active Feature Acquisition
• Learning ULMFiT and Self-Distillation with Calibration for Medical Dialogue System
• Scheduling Real-time Deep Learning Services as Imprecise Computations
• Deep Classifiers with Label Noise Modeling and Distance Awareness
• Selective Probabilistic Classifier Based on Hypothesis Testing
• Batch Inverse-Variance Weighting: Deep Heteroscedastic Regression
• Uncertainty Quantification For Low-Rank Matrix Completion With Heterogeneous and Sub-Exponential Noise
• Structured Ensembles: an Approach to Reduce the Memory Footprint of Ensemble Methods
• Acting upon Imagination: when to trust imagined trajectories in model based reinforcement learning
• Probabilistic Trust Intervals for Out of Distribution Detection
• Structured DropConnect for Uncertainty Inference in Image Classification
• Label Smoothed Embedding Hypothesis for Out-of-Distribution Detection
• Teaching Uncertainty Quantification in Machine Learning through Use Cases
• Video2Skill: Adapting Events in Demonstration Videos to Skills in an Environment using Cyclic MDP Homomorphisms
• Word-level confidence estimation for RNN transducers
• $\Delta$-UQ: Accurate Uncertainty Quantification via Anchor Marginalization
• ModelPred: A Framework for Predicting Trained Model from Training Data
• Scalable Consistency Training for Graph Neural Networks via Self-Ensemble Self-Distillation
• Machine-learned trends in mirror configurations in the Large Plasma Device
• Boosting Certified Robustness for Time Series Classification with Efficient Self-Ensemble