Kernel and Rich Regimes in Overparametrized Models

Abstract

A recent line of work studies overparametrized neural networks in the "kernel regime," i.e. when the network behaves during training as a kernelized linear predictor, and thus training with gradient descent has the effect of finding the minimum RKHS norm solution. This stands in contrast to other studies which demonstrate how gradient descent on overparametrized multilayer networks can induce rich implicit biases that are not RKHS norms. Building on an observation by Chizat and Bach, we show how the scale of the initialization controls the transition between the "kernel" (aka lazy) and "rich" (aka active) regimes and affects generalization properties in multilayer homogeneous models. We provide a complete and detailed analysis for a simple two-layer model that already exhibits an interesting and meaningful transition between the kernel and rich regimes, and we demonstrate the transition for more complex matrix factorization models and multilayer non-linear networks.

This paper has been substantially modified, updated, and expanded with additional content (arXiv:2002.09277). To avoid confusion with already existing citations, we are withdrawing the old version of this article

References in corpus (13)

• Guaranteed Minimum-Rank Solutions of Linear Matrix Equations via Nuclear Norm Minimization
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• A Mean Field View of the Landscape of Two-Layers Neural Networks
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• On Lazy Training in Differentiable Programming
• Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
• Characterizing Implicit Bias in Terms of Optimization Geometry
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• Gradient descent aligns the layers of deep linear networks
• Gradient Descent Maximizes the Margin of Homogeneous Neural Networks
• SGD Learns the Conjugate Kernel Class of the Network

Cited by in corpus (32)

• On the Opportunities and Risks of Foundation Models
• The Pitfalls of Simplicity Bias in Neural Networks
• Generalisation error in learning with random features and the hidden manifold model
• The large learning rate phase of deep learning: the catapult mechanism
• Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regime
• Classification vs regression in overparameterized regimes: Does the loss function matter?
• Bayesian Deep Ensembles via the Neural Tangent Kernel
• Frequency Bias in Neural Networks for Input of Non-Uniform Density
• The Implicit and Explicit Regularization Effects of Dropout
• On Learning Over-parameterized Neural Networks: A Functional Approximation Perspective
• Interpolated Adversarial Training: Achieving Robust Neural Networks without Sacrificing Too Much Accuracy
• Feature Learning in Infinite-Width Neural Networks
• Shape Matters: Understanding the Implicit Bias of the Noise Covariance
• On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layers
• When Does Preconditioning Help or Hurt Generalization?
• Towards Understanding Hierarchical Learning: Benefits of Neural Representations
• A Unified Paths Perspective for Pruning at Initialization
• On Infinite-Width Hypernetworks
• Mehler's Formula, Branching Process, and Compositional Kernels of Deep Neural Networks
• Implicit Regularization via Neural Feature Alignment
• Implicit Regularization in ReLU Networks with the Square Loss
• Shallow Univariate ReLu Networks as Splines: Initialization, Loss Surface, Hessian, & Gradient Flow Dynamics
• Approximation and Learning with Deep Convolutional Models: a Kernel Perspective
• Dynamically Stable Infinite-Width Limits of Neural Classifiers
• Extreme Memorization via Scale of Initialization
• Are wider nets better given the same number of parameters?
• An analytic theory of shallow networks dynamics for hinge loss classification
• On Margin Maximization in Linear and ReLU Networks
• From Boltzmann Machines to Neural Networks and Back Again
• Depth Without the Magic: Inductive Bias of Natural Gradient Descent
• Analytic expressions for the output evolution of a deep neural network
• Unintended Effects on Adaptive Learning Rate for Training Neural Network with Output Scale Change