On the linearity of large non-linear models: when and why the tangent kernel is constant

Abstract

The goal of this work is to shed light on the remarkable phenomenon of transition to linearity of certain neural networks as their width approaches infinity. We show that the transition to linearity of the model and, equivalently, constancy of the (neural) tangent kernel (NTK) result from the scaling properties of the norm of the Hessian matrix of the network as a function of the network width. We present a general framework for understanding the constancy of the tangent kernel via Hessian scaling applicable to the standard classes of neural networks. Our analysis provides a new perspective on the phenomenon of constant tangent kernel, which is different from the widely accepted "lazy training". Furthermore, we show that the transition to linearity is not a general property of wide neural networks and does not hold when the last layer of the network is non-linear. It is also not necessary for successful optimization by gradient descent.

arXiv commentaccepted as Spotlight in NeurIPS 2020; made correction to proof

References in corpus (8)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Searching for Activation Functions
• A Convergence Theory for Deep Learning via Over-Parameterization
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• On Lazy Training in Differentiable Programming
• Optimization for deep learning: theory and algorithms
• Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networks
• Any Target Function Exists in a Neighborhood of Any Sufficiently Wide Random Network: A Geometrical Perspective

Cited by in corpus (11)

• The Quantum Path Kernel: a Generalized Quantum Neural Tangent Kernel for Deep Quantum Machine Learning
• Wide and Deep Neural Networks Achieve Optimality for Classification
• On the Neural Tangent Kernel of Deep Networks with Orthogonal Initialization
• Loss landscapes and optimization in over-parameterized non-linear systems and neural networks
• An operator preconditioning perspective on training in physics-informed machine learning
• Score-based Generative Neural Networks for Large-Scale Optimal Transport
• Universal scaling laws in the gradient descent training of neural networks
• On the Global Convergence of Gradient Descent for multi-layer ResNets in the mean-field regime
• Subquadratic Overparameterization for Shallow Neural Networks
• A Modular Analysis of Provable Acceleration via Polyak's Momentum: Training a Wide ReLU Network and a Deep Linear Network
• Understanding Modern Techniques in Optimization: Frank-Wolfe, Nesterov's Momentum, and Polyak's Momentum