Gradient Descent Happens in a Tiny Subspace

Abstract

We show that in a variety of large-scale deep learning scenarios the gradient dynamically converges to a very small subspace after a short period of training. The subspace is spanned by a few top eigenvectors of the Hessian (equal to the number of classes in the dataset), and is mostly preserved over long periods of training. A simple argument then suggests that gradient descent may happen mostly in this subspace. We give an example of this effect in a solvable model of classification, and we comment on possible implications for optimization and learning.

9 pages + appendices, 12 figures

References in corpus (1)

• On the saddle point problem for non-convex optimization

Cited by in corpus (8)

• Scaling Laws for Neural Language Models
• An Empirical Model of Large-Batch Training
• The Early Phase of Neural Network Training
• Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians
• Negative eigenvalues of the Hessian in deep neural networks
• Subspace Inference for Bayesian Deep Learning
• How neural networks find generalizable solutions: Self-tuned annealing in deep learning
• On the Bias-Variance Tradeoff: Textbooks Need an Update