Low Dimensional Landscape Hypothesis is True: DNNs can be Trained in Tiny Subspaces
arXiv:2103.11154
Abstract
Deep neural networks (DNNs) usually contain massive parameters, but there is redundancy such that it is guessed that the DNNs could be trained in low-dimensional subspaces. In this paper, we propose a Dynamic Linear Dimensionality Reduction (DLDR) based on low-dimensional properties of the training trajectory. The reduction is efficient, which is supported by comprehensive experiments: optimization in 40 dimensional spaces can achieve comparable performance as regular training over thousands or even millions of parameters. Since there are only a few optimization variables, we develop a quasi-Newton-based algorithm and also obtain robustness against label noises, which are two follow-up experiments to show the advantages of finding low-dimensional subspaces.
References in corpus (13)
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Prototypical Networks for Few-shot Learning
- An overview of gradient descent optimization algorithms
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- A Convergence Theory for Deep Learning via Over-Parameterization
- RES: Regularized Stochastic BFGS Algorithm
- Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks
- Adaptive Bound Optimization for Online Convex Optimization
- Gradient Descent Happens in a Tiny Subspace
- Fast large-scale optimization by unifying stochastic gradient and quasi-Newton methods
- Practical Gauss-Newton Optimisation for Deep Learning
- Asymmetric Valleys: Beyond Sharp and Flat Local Minima
- Quasi-Newton's method in the class gradient defined high-curvature subspace