Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs

Abstract

The loss functions of deep neural networks are complex and their geometric properties are not well understood. We show that the optima of these complex loss functions are in fact connected by simple curves over which training and test accuracy are nearly constant. We introduce a training procedure to discover these high-accuracy pathways between modes. Inspired by this new geometric insight, we also propose a new ensembling method entitled Fast Geometric Ensembling (FGE). Using FGE we can train high-performing ensembles in the time required to train a single model. We achieve improved performance compared to the recent state-of-the-art Snapshot Ensembles, on CIFAR-10, CIFAR-100, and ImageNet.

arXiv commentAppears at Advances in Neural Information Processing Systems (NIPS), 2018

References in corpus (12)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Wide Residual Networks
• On Calibration of Modern Neural Networks
• SGDR: Stochastic Gradient Descent with Warm Restarts
• The Loss Surfaces of Multilayer Networks
• Visualizing the Loss Landscape of Neural Nets
• Averaging Weights Leads to Wider Optima and Better Generalization
• Qualitatively characterizing neural network optimization problems
• Horizontal and Vertical Ensemble with Deep Representation for Classification
• Exploring loss function topology with cyclical learning rates
• Using Mode Connectivity for Loss Landscape Analysis

Cited by in corpus (39)

• Lookahead Optimizer: k steps forward, 1 step back
• Averaging Weights Leads to Wider Optima and Better Generalization
• Fixing Data Augmentation to Improve Adversarial Robustness
• The Global Landscape of Neural Networks: An Overview
• Auto-Ensemble: An Adaptive Learning Rate Scheduling based Deep Learning Model Ensembling
• One-shot Federated Learning without Server-side Training
• Asymmetric Valleys: Beyond Sharp and Flat Local Minima
• The Unreasonable Effectiveness of Deep Evidential Regression
• Fed-ensemble: Improving Generalization through Model Ensembling in Federated Learning
• Policy Manifold Search: Exploring the Manifold Hypothesis for Diversity-based Neuroevolution
• FedBE: Making Bayesian Model Ensemble Applicable to Federated Learning
• Laplace HypoPINN: Physics-Informed Neural Network for hypocenter localization and its predictive uncertainty
• Connecting Low-Loss Subspace for Personalized Federated Learning
• Subspace Inference for Bayesian Deep Learning
• Some models are useful, but how do we know which ones? Towards a unified Bayesian model taxonomy
• On Last-Layer Algorithms for Classification: Decoupling Representation from Uncertainty Estimation
• Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimation
• Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape
• Robust, Accurate Stochastic Optimization for Variational Inference
• Selfish Sparse RNN Training
• Repulsive Deep Ensembles are Bayesian
• Generalization Bounds for Convolutional Neural Networks
• Wide-minima Density Hypothesis and the Explore-Exploit Learning Rate Schedule
• On Quantum Speedups for Nonconvex Optimization via Quantum Tunneling Walks
• Deep learning via message passing algorithms based on belief propagation
• On the impact of selected modern deep-learning techniques to the performance and celerity of classification models in an experimental high-energy physics use case
• Generating Neural Networks with Neural Networks
• How noise affects the Hessian spectrum in overparameterized neural networks
• On Connected Sublevel Sets in Deep Learning
• Using Mode Connectivity for Loss Landscape Analysis
• RRR-Net: Reusing, Reducing, and Recycling a Deep Backbone Network
• Pure and Spurious Critical Points: a Geometric Study of Linear Networks
• A Generative Model for Sampling High-Performance and Diverse Weights for Neural Networks
• The global optimum of shallow neural network is attained by ridgelet transform
• Extrapolation for Large-batch Training in Deep Learning
• Scalable Bayesian neural networks by layer-wise input augmentation
• Multivariate Deep Evidential Regression
• A study of local optima for learning feature interactions using neural networks
• Structured Ensembles: an Approach to Reduce the Memory Footprint of Ensemble Methods