A Walk with SGD

Abstract

We present novel empirical observations regarding how stochastic gradient descent (SGD) navigates the loss landscape of over-parametrized deep neural networks (DNNs). These observations expose the qualitatively different roles of learning rate and batch-size in DNN optimization and generalization. Specifically we study the DNN loss surface along the trajectory of SGD by interpolating the loss surface between parameters from consecutive \textit{iterations} and tracking various metrics during training. We find that the loss interpolation between parameters before and after each training iteration's update is roughly convex with a minimum (\textit{valley floor}) in between for most of the training. Based on this and other metrics, we deduce that for most of the training update steps, SGD moves in valley like regions of the loss surface by jumping from one valley wall to another at a height above the valley floor. This 'bouncing between walls at a height' mechanism helps SGD traverse larger distance for small batch sizes and large learning rates which we find play qualitatively different roles in the dynamics. While a large learning rate maintains a large height from the valley floor, a small batch size injects noise facilitating exploration. We find this mechanism is crucial for generalization because the valley floor has barriers and this exploration above the valley floor allows SGD to quickly travel far away from the initialization point (without being affected by barriers) and find flatter regions, corresponding to better generalization.

arXiv commentFirst two authors contributed equally

Cited by in corpus (16)

• A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay
• Don't Use Large Mini-Batches, Use Local SGD
• Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks
• Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks
• DeepFD: Automated Fault Diagnosis and Localization for Deep Learning Programs
• On the Computational Inefficiency of Large Batch Sizes for Stochastic Gradient Descent
• Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers
• Deep Gamblers: Learning to Abstain with Portfolio Theory
• The Implicit and Explicit Regularization Effects of Dropout
• LCA: Loss Change Allocation for Neural Network Training
• Wide-minima Density Hypothesis and the Explore-Exploit Learning Rate Schedule
• An Empirical Study of Large-Batch Stochastic Gradient Descent with Structured Covariance Noise
• Hessian based analysis of SGD for Deep Nets: Dynamics and Generalization
• LOSSGRAD: automatic learning rate in gradient descent
• Mini-batch Metropolis-Hastings MCMC with Reversible SGLD Proposal
• Parabolic Approximation Line Search for DNNs