Gradient Descent Happens in a Tiny Subspace

Abstract

We show that in a variety of large-scale deep learning scenarios the gradient dynamically converges to a very small subspace after a short period of training. The subspace is spanned by a few top eigenvectors of the Hessian (equal to the number of classes in the dataset), and is mostly preserved over long periods of training. A simple argument then suggests that gradient descent may happen mostly in this subspace. We give an example of this effect in a solvable model of classification, and we comment on possible implications for optimization and learning.

9 pages + appendices, 12 figures

References in corpus (5)

• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
• On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization
• Eigenvalues of the Hessian in Deep Learning: Singularity and Beyond
• On the saddle point problem for non-convex optimization

Cited by in corpus (60)

• Scaling Laws for Neural Language Models
• A Simple Baseline for Bayesian Uncertainty in Deep Learning
• An Empirical Model of Large-Batch Training
• Linear Mode Connectivity and the Lottery Ticket Hypothesis
• A Modern Take on the Bias-Variance Tradeoff in Neural Networks
• Understanding Gradient Clipping in Private SGD: A Geometric Perspective
• The Early Phase of Neural Network Training
• Pruning Neural Networks at Initialization: Why are We Missing the Mark?
• Asymptotics of Wide Networks from Feynman Diagrams
• The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD Training and Sample Size
• Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians
• A Diffusion Theory For Deep Learning Dynamics: Stochastic Gradient Descent Exponentially Favors Flat Minima
• Do Not Let Privacy Overbill Utility: Gradient Embedding Perturbation for Private Learning
• On the training dynamics of deep networks with $L_2$ regularization
• Negative eigenvalues of the Hessian in deep neural networks
• Subspace Inference for Bayesian Deep Learning
• Emergent properties of the local geometry of neural loss landscapes
• Accordion: Adaptive Gradient Communication via Critical Learning Regime Identification
• Bypassing the Ambient Dimension: Private SGD with Gradient Subspace Identification
• Large Scale Private Learning via Low-rank Reparametrization
• The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks
• Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networks
• Directional Pruning of Deep Neural Networks
• Federated Optimization of Smooth Loss Functions
• How noise affects the Hessian spectrum in overparameterized neural networks
• Provable Super-Convergence with a Large Cyclical Learning Rate
• SGD momentum optimizer with step estimation by online parabola model
• Learning Rates as a Function of Batch Size: A Random Matrix Theory Approach to Neural Network Training
• Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra
• Fast Dimension Independent Private AdaGrad on Publicly Estimated Subspaces
• Taxonomizing local versus global structure in neural network loss landscapes
• Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapes
• Improving Neural Network Training in Low Dimensional Random Bases
• The asymptotic spectrum of the Hessian of DNN throughout training
• Optimizing Neural Networks via Koopman Operator Theory
• Hessian based analysis of SGD for Deep Nets: Dynamics and Generalization
• AdaSGD: Bridging the gap between SGD and Adam
• How neural networks find generalizable solutions: Self-tuned annealing in deep learning
• Quasi-Newton's method in the class gradient defined high-curvature subspace
• The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusion
• Cockpit: A Practical Debugging Tool for the Training of Deep Neural Networks
• Does the Data Induce Capacity Control in Deep Learning?
• Public Data-Assisted Mirror Descent for Private Model Training
• Train-by-Reconnect: Decoupling Locations of Weights from their Values
• On the Bias-Variance Tradeoff: Textbooks Need an Update
• Low Dimensional Landscape Hypothesis is True: DNNs can be Trained in Tiny Subspaces
• RATT: Leveraging Unlabeled Data to Guarantee Generalization
• How many degrees of freedom do we need to train deep networks: a loss landscape perspective
• Intraclass clustering: an implicit learning ability that regularizes DNNs
• Privately Learning Subspaces
• Temporal mixture ensemble models for intraday volume forecasting in cryptocurrency exchange markets
• Quantum algorithm for finding the negative curvature direction in non-convex optimization
• Experiments with Rich Regime Training for Deep Learning
• ViViT: Curvature access through the generalized Gauss-Newton's low-rank structure
• A Framework for Private Matrix Analysis
• Structured Directional Pruning via Perturbation Orthogonal Projection
• A study on the plasticity of neural networks
• On the Impact of Stable Ranks in Deep Nets
• Analytic Study of Families of Spurious Minima in Two-Layer ReLU Neural Networks: A Tale of Symmetry II
• Subspace Adversarial Training