Training independent subnetworks for robust prediction

Abstract

Recent approaches to efficiently ensemble neural networks have shown that strong robustness and uncertainty performance can be achieved with a negligible gain in parameters over the original network. However, these methods still require multiple forward passes for prediction, leading to a significant computational cost. In this work, we show a surprising result: the benefits of using multiple predictions can be achieved `for free' under a single model's forward pass. In particular, we show that, using a multi-input multi-output (MIMO) configuration, one can utilize a single model's capacity to train multiple subnetworks that independently learn the task at hand. By ensembling the predictions made by the subnetworks, we improve model robustness without increasing compute. We observe a significant improvement in negative log-likelihood, accuracy, and calibration error on CIFAR10, CIFAR100, ImageNet, and their out-of-distribution variants compared to previous methods.

arXiv commentUpdated to the ICLR camera ready version, added reference to Soflaei et al. 2020

References in corpus (9)

• Pruning Convolutional Neural Networks for Resource Efficient Inference
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• Deep Ensembles: A Loss Landscape Perspective
• Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks
• Measuring the Effects of Data Parallelism on Neural Network Training
• BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning
• A Systematic Comparison of Bayesian Deep Learning Robustness in Diabetic Retinopathy Tasks
• Hydra: Preserving Ensemble Diversity for Model Distillation

Cited by in corpus (10)

• Giving Commands to a Self-Driving Car: How to Deal with Uncertain Situations?
• Shifts: A Dataset of Real Distributional Shift Across Multiple Large-Scale Tasks
• Collaboration of Experts: Achieving 80% Top-1 Accuracy on ImageNet with 100M FLOPs
• Improving robustness and calibration in ensembles with diversity regularization
• Mean Embeddings with Test-Time Data Augmentation for Ensembling of Representations
• Ex uno plures: Splitting One Model into an Ensemble of Subnetworks
• On Efficient Uncertainty Estimation for Resource-Constrained Mobile Applications
• Understanding Uncertainty in Bayesian Deep Learning
• Deep Classifiers with Label Noise Modeling and Distance Awareness
• Uncertainty Surrogates for Deep Learning