Using Mode Connectivity for Loss Landscape Analysis
arXiv:1806.06977
Abstract
Mode connectivity is a recently introduced frame- work that empirically establishes the connected- ness of minima by finding a high accuracy curve between two independently trained models. To investigate the limits of this setup, we examine the efficacy of this technique in extreme cases where the input models are trained or initialized differently. We find that the procedure is resilient to such changes. Given this finding, we propose using the framework for analyzing loss surfaces and training trajectories more generally, and in this direction, study SGD with cosine annealing and restarts (SGDR). We report that while SGDR moves over barriers in its trajectory, propositions claiming that it converges to and escapes from multiple local minima are not substantiated by our empirical results.
arXiv commentAccepted as a workshop paper at ICML's Workshop on Modern Trends in Nonconvex Optimization for Machine Learning, 2018
References in corpus (10)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Intriguing properties of neural networks
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- SGDR: Stochastic Gradient Descent with Warm Restarts
- On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
- The Marginal Value of Adaptive Gradient Methods in Machine Learning
- Averaging Weights Leads to Wider Optima and Better Generalization
- Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
- Essentially No Barriers in Neural Network Energy Landscape
Cited by in corpus (6)
- Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
- Multi-Representational Learning for Offline Signature Verification using Multi-Loss Snapshot Ensemble of CNNs
- Bridging Mode Connectivity in Loss Landscapes and Adversarial Robustness
- Loss Patterns of Neural Networks
- Directional Pruning of Deep Neural Networks
- Optimizing Mode Connectivity via Neuron Alignment