Snapshot Ensembles: Train 1, get M for free

Abstract

Ensembles of neural networks are known to be much more robust and accurate than individual networks. However, training multiple deep networks for model averaging is computationally expensive. In this paper, we propose a method to obtain the seemingly contradictory goal of ensembling multiple neural networks at no additional training cost. We achieve this goal by training a single neural network, converging to several local minima along its optimization path and saving the model parameters. To obtain repeated rapid convergence, we leverage recent work on cyclic learning rate schedules. The resulting technique, which we refer to as Snapshot Ensembling, is simple, yet surprisingly effective. We show in a series of experiments that our approach is compatible with diverse network architectures and learning tasks. It consistently yields lower error rates than state-of-the-art single models at no additional training cost, and compares favorably with traditional network ensembles. On CIFAR-10 and CIFAR-100 our DenseNet Snapshot Ensembles obtain error rates of 3.4% and 17.4% respectively.

Cited by in corpus (53)

• SGDR: Stochastic Gradient Descent with Warm Restarts
• LSTM Fully Convolutional Networks for Time Series Classification
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
• Don't Use Large Mini-Batches, Use Local SGD
• Simple And Efficient Architecture Search for Convolutional Neural Networks
• Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing
• How Does Learning Rate Decay Help Modern Neural Networks?
• An Ensemble of Epoch-wise Empirical Bayes for Few-shot Learning
• Big-Little Net: An Efficient Multi-Scale Feature Representation for Visual and Speech Recognition
• $\mathcal{G}$-softmax: Improving Intra-class Compactness and Inter-class Separability of Features
• OSLNet: Deep Small-Sample Classification with an Orthogonal Softmax Layer
• Language learning using Speech to Image retrieval
• Auto-Ensemble: An Adaptive Learning Rate Scheduling based Deep Learning Model Ensembling
• Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety
• Asymmetric Valleys: Beyond Sharp and Flat Local Minima
• Towards Better Accuracy-efficiency Trade-offs: Divide and Co-training
• Feature Space Singularity for Out-of-Distribution Detection
• Wisdom of Committees: An Overlooked Approach To Faster and More Accurate Models
• Augmenting correlation structures in spatial data using deep generative models
• PointAugment: an Auto-Augmentation Framework for Point Cloud Classification
• Neural Ensemble Search for Uncertainty Estimation and Dataset Shift
• Repulsive Deep Ensembles are Bayesian
• An Ensemble of Knowledge Sharing Models for Dynamic Hand Gesture Recognition
• Learning semantic sentence representations from visually grounded language without lexical knowledge
• MEAL: Multi-Model Ensemble via Adversarial Learning
• Learning Transferable Adversarial Examples via Ghost Networks
• Loss Patterns of Neural Networks
• On the impact of selected modern deep-learning techniques to the performance and celerity of classification models in an experimental high-energy physics use case
• Mirror Descent View for Neural Network Quantization
• Dense Uncertainty Estimation
• Exponential Moving Average Normalization for Self-supervised and Semi-supervised Learning
• On the Convergence of Stochastic Gradient Descent with Bandwidth-based Step Size
• On Stein Variational Neural Network Ensembles
• Collective Learning by Ensembles of Altruistic Diversifying Neural Networks
• Single-partition adaptive Q-learning
• Adaptive Focus for Efficient Video Recognition
• Efficient Evaluation-Time Uncertainty Estimation by Improved Distillation
• Auto-Annotation Quality Prediction for Semi-Supervised Learning with Ensembles
• Deep Networks with Fast Retraining
• SuperNet -- An efficient method of neural networks ensembling
• Neural Model-based Optimization with Right-Censored Observations
• Grad-CAM guided channel-spatial attention module for fine-grained visual classification
• Troubleshooting Blind Image Quality Models in the Wild
• Adversarial-Based Knowledge Distillation for Multi-Model Ensemble and Noisy Data Refinement
• Sharpen Focus: Learning with Attention Separability and Consistency
• Robust Deep Learning Ensemble against Deception
• How Does BN Increase Collapsed Neural Network Filters?
• Denoising and Verification Cross-Layer Ensemble Against Black-box Adversarial Attacks
• Boost Neural Networks by Checkpoints
• Single Model Ensemble using Pseudo-Tags and Distinct Vectors
• Pattern Recognition in Vital Signs Using Spectrograms
• n-MeRCI: A new Metric to Evaluate the Correlation Between Predictive Uncertainty and True Error
• Towards Comprehensive Monocular Depth Estimation: Multiple Heads Are Better Than One