Bad Global Minima Exist and SGD Can Reach Them

Abstract

Several works have aimed to explain why overparameterized neural networks generalize well when trained by Stochastic Gradient Descent (SGD). The consensus explanation that has emerged credits the randomized nature of SGD for the bias of the training process towards low-complexity models and, thus, for implicit regularization. We take a careful look at this explanation in the context of image classification with common deep neural network architectures. We find that if we do not regularize \emph{explicitly}, then SGD can be easily made to converge to poorly-generalizing, high-complexity models: all it takes is to first train on a random labeling on the data, before switching to properly training with the correct labels. In contrast, we find that in the presence of explicit regularization, pretraining with random labels has no detrimental effect on SGD. We believe that our results give evidence that explicit regularization plays a far more important role in the success of overparameterized neural networks than what has been understood until now. Specifically, by penalizing complicated models independently of their fit to the data, regularization affects training dynamics also far away from optima, making simple models that fit the data well discoverable by local methods, such as SGD.

References in corpus (15)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Explaining and Harnessing Adversarial Examples
• Understanding deep learning requires rethinking generalization
• Robustness May Be at Odds with Accuracy
• Sensitivity and Generalization in Neural Networks: an Empirical Study
• In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learning
• The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learning
• To understand deep learning we need to understand kernel learning
• Geometry of Optimization and Implicit Regularization in Deep Learning
• Implicit Regularization in Deep Learning
• Theory of Deep Learning III: explaining the non-overfitting puzzle
• Generalization Properties and Implicit Regularization for Multiple Passes SGM
• An Exponential Learning Rate Schedule for Deep Learning
• Minimum weight norm models do not always generalize well for over-parameterized problems
• Weak and Strong Gradient Directions: Explaining Memorization, Generalization, and Hardness of Examples at Scale

Cited by in corpus (12)

• Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problem
• Medical Image Classification Using Transfer Learning and Chaos Game Optimization on the Internet of Medical Things
• Multiplicative noise and heavy tails in stochastic optimization
• Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functions
• A consensus-based global optimization method for high dimensional machine learning problems
• Dynamical mean-field theory for stochastic gradient descent in Gaussian mixture classification
• Label Noise SGD Provably Prefers Flat Global Minimizers
• The Benefits of Implicit Regularization from SGD in Least Squares Problems
• Weak and Strong Gradient Directions: Explaining Memorization, Generalization, and Hardness of Examples at Scale
• Stochastic Gradient Descent outperforms Gradient Descent in recovering a high-dimensional signal in a glassy energy landscape
• On the security relevance of weights in deep learning
• Mental Models of Adversarial Machine Learning