Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent

Abstract

A longstanding goal in deep learning research has been to precisely characterize training and generalization. However, the often complex loss landscapes of neural networks have made a theory of learning dynamics elusive. In this work, we show that for wide neural networks the learning dynamics simplify considerably and that, in the infinite width limit, they are governed by a linear model obtained from the first-order Taylor expansion of the network around its initial parameters. Furthermore, mirroring the correspondence between wide Bayesian neural networks and Gaussian processes, gradient-based training of wide neural networks with a squared loss produces test set predictions drawn from a Gaussian process with a particular compositional kernel. While these theoretical results are only exact in the infinite width limit, we nevertheless find excellent empirical agreement between the predictions of the original network and those of the linearized version even for finite practically-sized networks. This agreement is robust across different architectures, optimization methods, and loss functions.

arXiv comment12+16 pages; open-source code available at https://github.com/google/neural-tangents; accepted to NeurIPS 2019

References in corpus (2)

• A Mean Field View of the Landscape of Two-Layers Neural Networks
• L2 Regularization versus Batch and Weight Normalization

Cited by in corpus (47)

• On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks
• Scaling description of generalization with number of parameters in deep learning
• Explaining Neural Scaling Laws
• The Modern Mathematics of Deep Learning
• Review: Deep Learning in Electron Microscopy
• Memorizing without overfitting: Bias, variance, and interpolation in over-parameterized models
• Neural Networks and Quantum Field Theory
• Unveiling the optimization process of Physics Informed Neural Networks: How accurate and competitive can PINNs be?
• Multi-scale Convolutional Neural Networks for Inverse Problems
• Fast and Sample-Efficient Interatomic Neural Network Potentials for Molecules and Materials Based on Gaussian Moments
• Learning Curves for Deep Neural Networks: A Gaussian Field Theory Perspective
• Fed-ensemble: Improving Generalization through Model Ensembling in Federated Learning
• Mathematical Models of Overparameterized Neural Networks
• Dimensionality Reduced Training by Pruning and Freezing Parts of a Deep Neural Network, a Survey
• Deep regularization and direct training of the inner layers of Neural Networks with Kernel Flows
• Wide and Deep Neural Networks Achieve Optimality for Classification
• Deep Q-learning: a robust control approach
• Locality defeats the curse of dimensionality in convolutional teacher-student scenarios
• Rethinking Influence Functions of Neural Networks in the Over-parameterized Regime
• Prediction rigidities for data-driven chemistry
• Laziness, Barren Plateau, and Noise in Machine Learning
• Metric Flows with Neural Networks
• Quantum-enhanced neural networks in the neural tangent kernel framework
• AIR-Net: Adaptive and Implicit Regularization Neural Network for Matrix Completion
• Competitive plasticity to reduce the energetic costs of learning
• Adaptive and Implicit Regularization for Matrix Completion
• Emergent Quantum Mechanics at the Boundary of a Local Classical Lattice Model
• The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusion
• Towards an Understanding of Residual Networks Using Neural Tangent Hierarchy (NTH)
• Bayesian RG Flow in Neural Network Field Theories
• Trained quantum neural networks are Gaussian processes
• LNPT: Label-free Network Pruning and Training
• Global Convergence of SGD On Two Layer Neural Nets
• Weight fluctuations in (deep) linear neural networks and a derivation of the inverse-variance flatness relation
• Quantitative convergence of trained quantum neural networks to a Gaussian process
• Uncertainty Quantification From Scaling Laws in Deep Neural Networks
• Effect of the initial configuration of weights on the training and function of artificial neural networks
• Rethinking generalization of classifiers in separable classes scenarios and over-parameterized regimes
• Peer-to-Peer Learning Dynamics of Wide Neural Networks
• Summary statistics of learning link changing neural representations to behavior
• Generalization Error Curves for Analytic Spectral Algorithms under Power-law Decay
• Generalisation under gradient descent via deterministic PAC-Bayes
• Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networks
• Catapult Dynamics and Phase Transitions in Quadratic Nets
• A note on regularised NTK dynamics with an application to PAC-Bayesian training
• Fast convergence to an approximate solution by message-passing for complex optimizations
• Transforming physics-informed machine learning to convex optimization