Asymmetric Valleys: Beyond Sharp and Flat Local Minima

Abstract

Despite the non-convex nature of their loss functions, deep neural networks are known to generalize well when optimized with stochastic gradient descent (SGD). Recent work conjectures that SGD with proper configuration is able to find wide and flat local minima, which have been proposed to be associated with good generalization performance. In this paper, we observe that local minima of modern deep networks are more than being flat or sharp. Specifically, at a local minimum there exist many asymmetric directions such that the loss increases abruptly along one side, and slowly along the opposite side--we formally define such minima as asymmetric valleys. Under mild assumptions, we prove that for asymmetric valleys, a solution biased towards the flat side generalizes better than the exact minimizer. Further, we show that simply averaging the weights along the SGD trajectory gives rise to such biased solutions implicitly. This provides a theoretical explanation for the intriguing phenomenon observed by Izmailov et al. (2018). In addition, we empirically find that batch normalization (BN) appears to be a major cause for asymmetric valleys.

submitted to ICML 2019

References in corpus (19)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Don't Decay the Learning Rate, Increase the Batch Size
• Making Gradient Descent Optimal for Strongly Convex Stochastic Optimization
• Revisiting Small Batch Training for Deep Neural Networks
• Three Factors Influencing Minima in SGD
• Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
• Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
• Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
• Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes
• Learning One-hidden-layer Neural Networks with Landscape Design
• Snapshot Ensembles: Train 1, get M for free
• Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
• Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approach
• Stronger generalization bounds for deep nets via a compression approach
• Risk and parameter convergence of logistic regression
• Neon2: Finding Local Minima via First-Order Oracles
• The loss landscape of overparameterized neural networks
• On the Local Minima of the Empirical Risk

Cited by in corpus (18)

• Optimization for deep learning: theory and algorithms
• Automatic Perturbation Analysis for Scalable Certified Robustness and Beyond
• A Diffusion Theory For Deep Learning Dynamics: Stochastic Gradient Descent Exponentially Favors Flat Minima
• Directional Pruning of Deep Neural Networks
• Information-Theoretic Generalization Bounds for Stochastic Gradient Descent
• Dynamic of Stochastic Gradient Descent with State-Dependent Noise
• Hierarchical Weight Averaging for Deep Neural Networks
• Taxonomizing local versus global structure in neural network loss landscapes
• Improving Neural Network Training in Low Dimensional Random Bases
• Formal Language Theory Meets Modern NLP
• Deep Curvature Suite
• Low Dimensional Landscape Hypothesis is True: DNNs can be Trained in Tiny Subspaces
• AlterSGD: Finding Flat Minima for Continual Learning by Alternative Training
• Embedding Principle of Loss Landscape of Deep Neural Networks
• Embedding Principle in Depth for the Loss Landscape Analysis of Deep Neural Networks
• Generalisation under gradient descent via deterministic PAC-Bayes
• Embedding Principle: a hierarchical structure of loss landscape of deep neural networks
• Parabolic Approximation Line Search for DNNs