On the Relation Between the Sharpest Directions of DNN Loss and the SGD Step Length

Abstract

Stochastic Gradient Descent (SGD) based training of neural networks with a large learning rate or a small batch-size typically ends in well-generalizing, flat regions of the weight space, as indicated by small eigenvalues of the Hessian of the training loss. However, the curvature along the SGD trajectory is poorly understood. An empirical investigation shows that initially SGD visits increasingly sharp regions, reaching a maximum sharpness determined by both the learning rate and the batch-size of SGD. When studying the SGD dynamics in relation to the sharpest directions in this initial phase, we find that the SGD step is large compared to the curvature and commonly fails to minimize the loss along the sharpest directions. Furthermore, using a reduced learning rate along these directions can improve training speed while leading to both sharper and better generalizing solutions compared to vanilla SGD. In summary, our analysis of the dynamics of SGD in the subspace of the sharpest directions shows that they influence the regions that SGD steers to (where larger learning rate or smaller batch size result in wider regions visited), the overall training speed, and the generalization ability of the final model.

Cited by in corpus (23)

• When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations
• Understanding the Role of Training Regimes in Continual Learning
• Stiffness: A New Perspective on Generalization in Neural Networks
• The Break-Even Point on Optimization Trajectories of Deep Neural Networks
• Laplacian Smoothing Gradient Descent
• The Implicit and Explicit Regularization Effects of Dropout
• Consensus Control for Decentralized Deep Learning
• Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability
• A Closer Look at Structured Pruning for Neural Network Compression
• LCA: Loss Change Allocation for Neural Network Training
• Accordion: Adaptive Gradient Communication via Critical Learning Regime Identification
• Wide-minima Density Hypothesis and the Explore-Exploit Learning Rate Schedule
• On Learning Rates and Schrödinger Operators
• Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networks
• Learning Rate Annealing Can Provably Help Generalization, Even for Convex Problems
• Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra
• S-SGD: Symmetrical Stochastic Gradient Descent with Weight Noise Injection for Reaching Flat Minima
• Unifying Regularisation Methods for Continual Learning
• The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusion
• Gradient Noise Convolution (GNC): Smoothing Loss Function for Distributed Large-Batch SGD
• Critical Learning Periods in Federated Learning
• Analytic expressions for the output evolution of a deep neural network
• Rapidly Adapting Moment Estimation