On Last-Layer Algorithms for Classification: Decoupling Representation from Uncertainty Estimation

Abstract

Uncertainty quantification for deep learning is a challenging open problem. Bayesian statistics offer a mathematically grounded framework to reason about uncertainties; however, approximate posteriors for modern neural networks still require prohibitive computational costs. We propose a family of algorithms which split the classification task into two stages: representation learning and uncertainty estimation. We compare four specific instances, where uncertainty estimation is performed via either an ensemble of Stochastic Gradient Descent or Stochastic Gradient Langevin Dynamics snapshots, an ensemble of bootstrapped logistic regressions, or via a number of Monte Carlo Dropout passes. We evaluate their performance in terms of \emph{selective} classification (risk-coverage), and their ability to detect out-of-distribution samples. Our experiments suggest there is limited value in adding multiple uncertainty layers to deep classifiers, and we observe that these simple methods strongly outperform a vanilla point-estimate SGD in some complex benchmarks like ImageNet.

References in corpus (25)

• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• How transferable are features in deep neural networks?
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
• A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
• Stochastic Gradient Hamiltonian Monte Carlo
• Scalable Bayesian Optimization Using Deep Neural Networks
• Deep Anomaly Detection with Outlier Exposure
• Learning Confidence for Out-of-Distribution Detection in Neural Networks
• Deep and Confident Prediction for Time Series at Uber
• Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
• Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks
• User-friendly guarantees for the Langevin Monte Carlo with inaccurate gradient
• Generative Probabilistic Novelty Detection with Adversarial Autoencoders
• Bayesian inference and the parametric bootstrap
• Bayesian Posterior Sampling via Stochastic Gradient Fisher Scoring
• Evaluating Uncertainty Quantification in End-to-End Autonomous Driving Control
• The promises and pitfalls of Stochastic Gradient Langevin Dynamics
• Improving Output Uncertainty Estimation and Generalization in Deep Learning via Neural Network Gaussian Processes
• Bias-Reduced Uncertainty Estimation for Deep Neural Classifiers
• The True Cost of Stochastic Gradient Langevin Dynamics
• Out-of-Distribution Detection using Multiple Semantic Label Representations
• The Relevance of Bayesian Layer Positioning to Model Uncertainty in Deep Bayesian Active Learning
• A Less Biased Evaluation of Out-of-distribution Sample Detectors
• Structured Output Learning with Abstention: Application to Accurate Opinion Prediction

Cited by in corpus (8)

• Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users
• Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks
• Fast Predictive Uncertainty for Classification with Bayesian Deep Networks
• Deep Ensembles from a Bayesian Perspective
• Bayesian Neural Networks: Essentials
• Robust Semantic Segmentation with Superpixel-Mix
• Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learning
• Neural density estimation and uncertainty quantification for laser induced breakdown spectroscopy spectra