On Lazy Training in Differentiable Programming

Abstract

In a series of recent theoretical works, it was shown that strongly over-parameterized neural networks trained with gradient-based methods could converge exponentially fast to zero training loss, with their parameters hardly varying. In this work, we show that this "lazy training" phenomenon is not specific to over-parameterized neural networks, and is due to a choice of scaling, often implicit, that makes the model behave as its linearization around the initialization, thus yielding a model equivalent to learning with positive-definite kernels. Through a theoretical analysis, we exhibit various situations where this phenomenon arises in non-convex optimization and we provide bounds on the distance between the lazy and linearized optimization paths. Our numerical experiments bring a critical note, as we observe that the performance of commonly used non-linear deep convolutional neural networks in computer vision degrades when trained in the lazy regime. This makes it unlikely that "lazy training" is behind the many successes of neural networks in difficult high dimensional tasks.

Cited by in corpus (158)

• Underspecification Presents Challenges for Credibility in Modern Machine Learning
• The Principles of Deep Learning Theory
• Scaling description of generalization with number of parameters in deep learning
• Explaining Neural Scaling Laws
• Optimization for deep learning: theory and algorithms
• The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies
• How Neural Networks Extrapolate: From Feedforward to Graph Neural Networks
• Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth
• Self-Distillation Amplifies Regularization in Hilbert Space
• A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics
• A comprehensive review of Quantum Machine Learning: from NISQ to Fault Tolerance
• The Pitfalls of Simplicity Bias in Neural Networks
• Generalisation error in learning with random features and the hidden manifold model
• Representation Learning via Quantum Neural Tangent Kernels
• The Global Landscape of Neural Networks: An Overview
• Towards Understanding the Spectral Bias of Deep Learning
• The large learning rate phase of deep learning: the catapult mechanism
• Kernel and Rich Regimes in Overparametrized Models
• Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regime
• Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspective
• Explaining the physics of transfer learning a data-driven subgrid-scale closure to a different turbulent flow
• The Gaussian equivalence of generative models for learning with shallow neural networks
• On the linearity of large non-linear models: when and why the tangent kernel is constant
• A Geometric Analysis of Neural Collapse with Unconstrained Features
• Fast and Sample-Efficient Interatomic Neural Network Potentials for Molecules and Materials Based on Gaussian Moments
• Finite Versus Infinite Neural Networks: an Empirical Study
• Explicit Regularisation in Gaussian Noise Injections
• Multiplicative noise and heavy tails in stochastic optimization
• Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeed
• A Generalized Neural Tangent Kernel Analysis for Two-layer Neural Networks
• Frequency Bias in Neural Networks for Input of Non-Uniform Density
• Bayesian Deep Ensembles via the Neural Tangent Kernel
• Triple descent and the two kinds of overfitting: Where & why do they appear?
• How Much Over-parameterization Is Sufficient to Learn Deep ReLU Networks?
• Mathematical Models of Overparameterized Neural Networks
• The Quantum Path Kernel: a Generalized Quantum Neural Tangent Kernel for Deep Quantum Machine Learning
• Convergence analysis for gradient flows in the training of artificial neural networks with ReLU activation
• A Priori Generalization Analysis of the Deep Ritz Method for Solving High Dimensional Elliptic Equations
• Random Features for Kernel Approximation: A Survey on Algorithms, Theory, and Beyond
• Asymptotics of Ridge (less) Regression under General Source Condition
• Network size and weights size for memorization with two-layers neural networks
• A Random Matrix Analysis of Random Fourier Features: Beyond the Gaussian Kernel, a Precise Phase Transition, and the Corresponding Double Descent
• Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networks
• Selective Amnesia: On Efficient, High-Fidelity and Blind Suppression of Backdoor Effects in Trojaned Machine Learning Models
• Locality defeats the curse of dimensionality in convolutional teacher-student scenarios
• Accelerated replica exchange stochastic gradient Langevin diffusion enhanced Bayesian DeepONet for solving noisy parametric PDEs
• On the existence of global minima and convergence analyses for gradient descent methods in the training of deep neural networks
• On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layers
• On the Neural Tangent Kernel of Deep Networks with Orthogonal Initialization
• Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learning
• Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstruction
• WeMix: How to Better Utilize Data Augmentation
• Taylorized Training: Towards Better Approximation of Neural Network Training at Finite Width
• Temperature check: theory and practice for training models with softmax-cross-entropy losses
• The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks
• On the Convergence of Gradient Descent Training for Two-layer ReLU-networks in the Mean Field Regime
• A Unifying View on Implicit Bias in Training Linear Neural Networks
• Understanding Overparameterization in Generative Adversarial Networks
• A semigroup method for high dimensional committor functions based on neural network
• Quantum Lazy Training
• Laziness, Barren Plateau, and Noise in Machine Learning
• Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regime
• Large-time asymptotics in deep learning
• A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learning
• Towards Understanding Hierarchical Learning: Benefits of Neural Representations
• Deep Equals Shallow for ReLU Networks in Kernel Regimes
• On the Global Convergence of Training Deep Linear ResNets
• How Powerful are Shallow Neural Networks with Bandlimited Random Weights?
• Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracy
• On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descent
• On the interplay between data structure and loss function in classification problems
• Understanding and Accelerating Neural Architecture Search with Training-Free and Theory-Grounded Metrics
• Global Convergence of Second-order Dynamics in Two-layer Neural Networks
• Richer priors for infinitely wide multi-layer perceptrons
• Relative stability toward diffeomorphisms indicates performance in deep nets
• Existence, uniqueness, and convergence rates for gradient flows in the training of artificial neural networks with ReLU activation
• When does gradient descent with logistic loss interpolate using deep networks with smoothed ReLU activations?
• Shallow Univariate ReLu Networks as Splines: Initialization, Loss Surface, Hessian, & Gradient Flow Dynamics
• Approximation and Learning with Deep Convolutional Models: a Kernel Perspective
• Generalization Guarantees for Neural Architecture Search with Train-Validation Split
• Fast Learning in Reproducing Kernel Krein Spaces via Signed Measures
• Exploring Weight Importance and Hessian Bias in Model Pruning
• Non-asymptotic approximations of neural networks by Gaussian processes
• Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural Networks
• Neural Spectrahedra and Semidefinite Lifts: Global Convex Optimization of Polynomial Activation Neural Networks in Fully Polynomial-Time
• The Nonlinearity Coefficient - A Practical Guide to Neural Architecture Design
• Quantifying the Benefit of Using Differentiable Learning over Tangent Kernels
• Meta-learning with negative learning rates
• Neural Tangent Kernel Maximum Mean Discrepancy
• A Recipe for Global Convergence Guarantee in Deep Neural Networks
• On the Double Descent of Random Features Models Trained with SGD
• Finding trainable sparse networks through Neural Tangent Transfer
• Global Convergence of Three-layer Neural Networks in the Mean Field Regime
• Universal scaling laws in the gradient descent training of neural networks
• Learning with Gradient Descent and Weakly Convex Losses
• Wide Neural Networks Forget Less Catastrophically
• When does gradient descent with logistic loss find interpolating two-layer networks?
• Perspective: A Phase Diagram for Deep Learning unifying Jamming, Feature Learning and Lazy Training
• A Dynamical View on Optimization Algorithms of Overparameterized Neural Networks
• Does the Data Induce Capacity Control in Deep Learning?
• Which Minimizer Does My Neural Network Converge To?
• Temporal-difference learning with nonlinear function approximation: lazy training and mean field regimes
• Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticity
• Training Two-Layer ReLU Networks with Gradient Descent is Inconsistent
• Kernel-Based Smoothness Analysis of Residual Networks
• Plateau Phenomenon in Gradient Descent Training of ReLU networks: Explanation, Quantification and Avoidance
• Global Convergence of Deep Networks with One Wide Layer Followed by Pyramidal Topology
• Implicit bias of deep linear networks in the large learning rate phase
• What Happens after SGD Reaches Zero Loss? --A Mathematical Framework
• An Optimization and Generalization Analysis for Max-Pooling Networks
• On the Empirical Neural Tangent Kernel of Standard Finite-Width Convolutional Neural Network Architectures
• A finite sample analysis of the benign overfitting phenomenon for ridge function estimation
• Frank-Wolfe Methods in Probability Space
• Analyzing Finite Neural Networks: Can We Trust Neural Tangent Kernel Theory?
• Label-Aware Neural Tangent Kernel: Toward Better Generalization and Local Elasticity
• The Dynamics of Gradient Descent for Overparametrized Neural Networks
• The Unreasonable Effectiveness of Patches in Deep Convolutional Kernels Methods
• Nonasymptotic Guarantees for Spiked Matrix Recovery with Generative Priors
• An analytic theory of shallow networks dynamics for hinge loss classification
• Scaling Neural Tangent Kernels via Sketching and Random Features
• RATT: Leveraging Unlabeled Data to Guarantee Generalization
• Neural Networks as Kernel Learners: The Silent Alignment Effect
• Finite-Time Consensus Learning for Decentralized Optimization with Nonlinear Gossiping
• Deformed semicircle law and concentration of nonlinear random matrices for ultra-wide neural networks
• Understanding Deflation Process in Over-parametrized Tensor Decomposition
• An Even More Optimal Stochastic Optimization Algorithm: Minibatching and Interpolation Learning
• Properties of the After Kernel
• Effect of the initial configuration of weights on the training and function of artificial neural networks
• Provably Training Overparameterized Neural Network Classifiers with Non-convex Constraints
• Tight Bounds on the Smallest Eigenvalue of the Neural Tangent Kernel for Deep ReLU Networks
• Generalization error of minimum weighted norm and kernel interpolation
• Experiments with Rich Regime Training for Deep Learning
• Why is AI hard and Physics simple?
• Spectral Analysis of the Neural Tangent Kernel for Deep Residual Networks
• Predicting the outputs of finite deep neural networks trained with noisy gradients
• Towards a General Theory of Infinite-Width Limits of Neural Classifiers
• Rethinking generalization of classifiers in separable classes scenarios and over-parameterized regimes
• A Modular Analysis of Provable Acceleration via Polyak's Momentum: Training a Wide ReLU Network and a Deep Linear Network
• Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networks
• Over-Parametrized Matrix Factorization in the Presence of Spurious Stationary Points
• Over-parametrized neural networks as under-determined linear systems
• Global optimality of softmax policy gradient with single hidden layer neural networks in the mean-field regime
• Avoiding Kernel Fixed Points: Computing with ELU and GELU Infinite Networks
• Uniform Generalization Bounds for Overparameterized Neural Networks
• Classification and Adversarial examples in an Overparameterized Linear Model: A Signal Processing Perspective
• Unintended Effects on Adaptive Learning Rate for Training Neural Network with Output Scale Change
• One-pass Stochastic Gradient Descent in Overparametrized Two-layer Neural Networks
• Training Linear Neural Networks: Non-Local Convergence and Complexity Results
• Generalization Properties of hyper-RKHS and its Applications
• Why Does Multi-Epoch Training Help?
• Understanding Modern Techniques in Optimization: Frank-Wolfe, Nesterov's Momentum, and Polyak's Momentum
• Nearly Minimal Over-Parametrization of Shallow Neural Networks
• Dual Training of Energy-Based Models with Overparametrized Shallow Neural Networks
• Neural Contextual Bandits without Regret
• Analytic expressions for the output evolution of a deep neural network
• Random Features for the Neural Tangent Kernel
• The Probabilistic Fault Tolerance of Neural Networks in the Continuous Limit
• Provable Generalization of SGD-trained Neural Networks of Any Width in the Presence of Adversarial Label Noise