Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity

Abstract

The success of deep ensembles on improving predictive performance, uncertainty estimation, and out-of-distribution robustness has been extensively studied in the machine learning literature. Albeit the promising results, naively training multiple deep neural networks and combining their predictions at inference leads to prohibitive computational costs and memory requirements. Recently proposed efficient ensemble approaches reach the performance of the traditional deep ensembles with significantly lower costs. However, the training resources required by these approaches are still at least the same as training a single dense model. In this work, we draw a unique connection between sparse neural network training and deep ensembles, yielding a novel efficient ensemble learning framework called FreeTickets. Instead of training multiple dense networks and averaging them, we directly train sparse subnetworks from scratch and extract diverse yet accurate subnetworks during this efficient, sparse-to-sparse training. Our framework, FreeTickets, is defined as the ensemble of these relatively cheap sparse subnetworks. Despite being an ensemble method, FreeTickets has even fewer parameters and training FLOPs than a single dense model. This seemingly counter-intuitive outcome is due to the ultra training/inference efficiency of dynamic sparse training. FreeTickets surpasses the dense baseline in all the following criteria: prediction accuracy, uncertainty estimation, out-of-distribution (OoD) robustness, as well as efficiency for both training and inference. Impressively, FreeTickets outperforms the naive deep ensemble with ResNet50 on ImageNet using around only 1/5 of the training FLOPs required by the latter. We have released our source code at https://github.com/VITA-Group/FreeTickets.

published in International Conference on Learning Representations (ICLR 2022)

References in corpus (29)

• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science
• The State of Sparsity in Deep Neural Networks
• Deep Ensembles: A Loss Landscape Perspective
• Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks
• Sparse Networks from Scratch: Faster Training without Losing Performance
• Natural Adversarial Examples
• Comparing Rewinding and Fine-tuning in Neural Network Pruning
• Deep Rewiring: Training very sparse deep networks
• Picking Winning Tickets Before Training by Preserving Gradient Flow
• Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization
• One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers
• Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP
• Chasing Sparsity in Vision Transformers: An End-to-End Exploration
• A topological insight into restricted Boltzmann machines
• Ensemble Methods as a Defense to Adversarial Perturbations Against Deep Neural Networks
• Fixing the train-test resolution discrepancy: FixEfficientNet
• SpaceNet: Make Free Space For Continual Learning
• The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models
• The Difficulty of Training Sparse Neural Networks
• Top-KAST: Top-K Always Sparse Training
• Sparse Transfer Learning via Winning Lottery Tickets
• Selfish Sparse RNN Training
• Keep the Gradients Flowing: Using Gradient Flow to Study Sparse Network Optimization
• Sparse Weight Activation Training

Cited by in corpus (2)

• Deep Isolation Forest for Anomaly Detection
• Uncertainty-Aware Rank-One MIMO Q Network Framework for Accelerated Offline Reinforcement Learning