Learning Neural Network Subspaces
arXiv:2102.10472
Abstract
Recent observations have advanced our understanding of the neural network optimization landscape, revealing the existence of (1) paths of high accuracy containing diverse solutions and (2) wider minima offering improved performance. Previous methods observing diverse paths require multiple training runs. In contrast we aim to leverage both property (1) and (2) with a single method and in a single training run. With a similar computational cost as training one model, we learn lines, curves, and simplexes of high-accuracy neural networks. These neural network subspaces contain diverse solutions that can be ensembled, approaching the ensemble performance of independently trained networks without the training cost. Moreover, using the subspace midpoint boosts accuracy, calibration, and robustness to label noise, outperforming Stochastic Weight Averaging.
References in corpus (20)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
- Wide Residual Networks
- On Calibration of Modern Neural Networks
- SGDR: Stochastic Gradient Descent with Warm Restarts
- Weight Uncertainty in Neural Networks
- When Does Label Smoothing Help?
- Underspecification Presents Challenges for Credibility in Modern Machine Learning
- Deep Ensembles: A Loss Landscape Perspective
- Averaging Weights Leads to Wider Optima and Better Generalization
- Essentially No Barriers in Neural Network Energy Landscape
- A Simple Baseline for Bayesian Uncertainty in Deep Learning
- Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning
- Sharpness-Aware Minimization for Efficiently Improving Generalization
- Pruning neural networks without any data by iteratively conserving synaptic flow
- Measuring the Intrinsic Dimension of Objective Landscapes
- The Difficulty of Training Sparse Neural Networks
- Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel
- Revisiting "Qualitatively Characterizing Neural Network Optimization Problems"