Neural Bootstrapper

Abstract

Bootstrapping has been a primary tool for ensemble and uncertainty quantification in machine learning and statistics. However, due to its nature of multiple training and resampling, bootstrapping deep neural networks is computationally burdensome; hence it has difficulties in practical application to the uncertainty estimation and related tasks. To overcome this computational bottleneck, we propose a novel approach called \emph{Neural Bootstrapper} (NeuBoots), which learns to generate bootstrapped neural networks through single model training. NeuBoots injects the bootstrap weights into the high-level feature layers of the backbone network and outputs the bootstrapped predictions of the target, without additional parameters and the repetitive computations from scratch. We apply NeuBoots to various machine learning tasks related to uncertainty quantification, including prediction calibrations in image classification and semantic segmentation, active learning, and detection of out-of-distribution samples. Our empirical results show that NeuBoots outperforms other bagging based methods under a much lower computational cost without losing the validity of bootstrapping.

arXiv comment19 pages, 13 figures. Accepted for NeurIPS 2021. Corresponding Author: Sungbin Lim

References in corpus (12)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Rethinking Atrous Convolution for Semantic Image Segmentation
• A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
• A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• Training Deep Neural Networks on Noisy Labels with Bootstrapping
• Accurate Uncertainties for Deep Learning Using Calibrated Regression
• Deep Ensembles: A Loss Landscape Perspective
• Deep Confidence: A Computationally Efficient Framework for Calculating Reliable Errors for Deep Neural Networks
• Scalable Neural Architecture Search for 3D Medical Image Segmentation
• Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning
• Confidence-Aware Learning for Deep Neural Networks