The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks
arXiv:2006.14599
Abstract
Modern neural networks are often regarded as complex black-box functions whose behavior is difficult to understand owing to their nonlinear dependence on the data and the nonconvexity in their loss landscapes. In this work, we show that these common perceptions can be completely false in the early phase of learning. In particular, we formally prove that, for a class of well-behaved input distributions, the early-time learning dynamics of a two-layer fully-connected neural network can be mimicked by training a simple linear model on the inputs. We additionally argue that this surprising simplicity can persist in networks with more layers and with convolutional architecture, which we verify empirically. Key to our analysis is to bound the spectral norm of the difference between the Neural Tangent Kernel (NTK) at initialization and an affine transform of the data kernel; however, unlike many previous results utilizing the NTK, we do not require the network to have disproportionately large width, and the network is allowed to escape the kernel regime later in training.
References in corpus (19)
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
- Towards a Human-like Open-Domain Chatbot
- Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
- On Lazy Training in Differentiable Programming
- Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
- Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
- On the Spectral Bias of Neural Networks
- Gradient Descent Happens in a Tiny Subspace
- Gradient descent aligns the layers of deep linear networks
- Enhanced Convolutional Neural Tangent Kernels
- Implicit Regularization in Deep Learning May Not Be Explainable by Norms
- Critical Learning Periods in Deep Neural Networks
- The large learning rate phase of deep learning: the catapult mechanism
- Neural Tangents: Fast and Easy Infinite Neural Networks in Python
- An analytic theory of generalization dynamics and transfer learning in deep linear networks
- SGD on Neural Networks Learns Functions of Increasing Complexity
- A Fine-Grained Spectral Perspective on Neural Networks
- Understanding training and generalization in deep learning by Fourier analysis