The Training Process of Many Deep Networks Explores the Same Low-Dimensional Manifold
arXiv:2305.01604 · doi:10.1073/pnas.2310002121
Abstract
We develop information-geometric techniques to analyze the trajectories of the predictions of deep networks during training. By examining the underlying high-dimensional probabilistic models, we reveal that the training process explores an effectively low-dimensional manifold. Networks with a wide range of architectures, sizes, trained using different optimization methods, regularization techniques, data augmentation techniques, and weight initializations lie on the same manifold in the prediction space. We study the details of this manifold to find that networks with different architectures follow distinguishable trajectories but other factors have a minimal influence; larger networks train along a similar manifold as that of smaller networks, just faster; and networks initialized at very different parts of the prediction space converge to the solution along a similar manifold.
References in corpus (23)
- Adam: A Method for Stochastic Optimization
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Distilling the Knowledge in a Neural Network
- Reconciling modern machine learning practice and the bias-variance trade-off
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- Understanding deep learning requires rethinking generalization
- A mathematical theory of semantic development in deep neural networks
- Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
- The geometry of nonlinear least squares with applications to sloppy models and optimization
- Spectrally-normalized margin bounds for neural networks
- The Burbea-Rao and Bhattacharyya centroids
- An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
- Just Interpolate: Kernel "Ridgeless" Regression Can Generalize
- Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit
- AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights
- A Purely Algebraic Justification of the Kabsch-Umeyama Algorithm
- Visualizing probabilistic models: Intensive Principal Component Analysis
- Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeed
- Visualizing probabilistic models in Minkowski space with intensive symmetrized Kullback-Leibler embedding
- PCA of high dimensional random walks with comparison to neural network training
- On the interplay between data structure and loss function in classification problems
- Does the Data Induce Capacity Control in Deep Learning?