Scaling description of generalization with number of parameters in deep learning

Abstract

Supervised deep learning involves the training of neural networks with a large number $N$ of parameters. For large enough $N$, in the so-called over-parametrized regime, one can essentially fit the training data points. Sparsity-based arguments would suggest that the generalization error increases as $N$ grows past a certain threshold $N^{\ast }$. Instead, empirical studies have shown that in the over-parametrized regime, generalization error keeps decreasing with $N$. We resolve this paradox through a new framework. We rely on the so-called Neural Tangent Kernel, which connects large neural nets to kernel methods, to show that the initialization causes finite-size random fluctuations $\Vert f_N-{\bar{f}}_N\Vert \sim N^{-1/4}$ of the neural net output function $f_N$ around its expectation ${\bar{f}}_N$. These affect the generalization error ${\epsilon }_N$ for classification: under natural assumptions, it decays to a plateau value ${\epsilon }_{\infty }$ in a power-law fashion $\sim N^{-1/2}$. This description breaks down at a so-called jamming transition $N=N^{\ast }$. At this threshold, we argue that $\Vert f_N\Vert$ diverges. This result leads to a plausible explanation for the cusp in test error known to occur at $N^{\ast }$. Our results are confirmed by extensive empirical observations on the MNIST and CIFAR image datasets. Our analysis finally suggests that, given a computational envelope, the smallest generalization error is obtained using several networks of intermediate sizes, just beyond $N^{\ast }$, and averaging their outputs.

arXiv commentThe clarity of the text has been improved: the section "Related works" has been updated and the section "3.1 Regression task" has been added

References in corpus (34)

• Adam: A Method for Stochastic Optimization
• Sequence to Sequence Learning with Neural Networks
• Reconciling modern machine learning practice and the bias-variance trade-off
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• The Loss Surfaces of Multilayer Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Deep Neural Networks as Gaussian Processes
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• Train longer, generalize better: closing the generalization gap in large batch training of neural networks
• On Exact Computation with an Infinitely Wide Neural Net
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent
• On Lazy Training in Differentiable Programming
• The generalization error of random features regression: Precise asymptotics and double descent curve
• Universal Spectrum of Normal Modes in Low-Temperature Glasses: an Exact Solution
• Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
• Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
• No bad local minima: Data independent training error guarantees for multilayer neural networks
• The simplest model of jamming
• Universality of the SAT-UNSAT (jamming) threshold in non-convex continuous constraint satisfaction problems
• The jamming transition as a paradigm to understand the loss landscape of deep neural networks
• Perspective: Energy Landscapes for Machine Learning
• Geometry of Optimization and Implicit Regularization in Deep Learning
• A jamming transition from under- to over-parametrization affects loss landscape and generalization
• A Modern Take on the Bias-Variance Tradeoff in Neural Networks
• Surprises in High-Dimensional Ridgeless Least Squares Interpolation
• Jamming in multilayer supervised learning models
• The loss landscape of overparameterized neural networks
• Mean Field Limit of the Learning Dynamics of Multilayer Neural Networks
• The Dynamics of Learning: A Random Matrix Approach
• Finite Depth and Width Corrections to the Neural Tangent Kernel
• Minnorm training: an algorithm for training over-parameterized deep neural networks
• Stuck in a What? Adventures in Weight Space

Cited by in corpus (68)

• Scaling Laws for Neural Language Models
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• Evaluation of the MACE Force Field Architecture: from Medicinal Chemistry to Materials Science
• Explaining Neural Scaling Laws
• The Modern Mathematics of Deep Learning
• The jamming transition as a paradigm to understand the loss landscape of deep neural networks
• Review: Deep Learning in Electron Microscopy
• Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit
• The generalization error of max-margin linear classifiers: Benign overfitting and high dimensional asymptotics in the overparametrized regime
• Generalisation error in learning with random features and the hidden manifold model
• Memorizing without overfitting: Bias, variance, and interpolation in over-parameterized models
• A Modern Take on the Bias-Variance Tradeoff in Neural Networks
• Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regime
• Optimal Regularization Can Mitigate Double Descent
• Asymptotics of Wide Networks from Feynman Diagrams
• Finite Versus Infinite Neural Networks: an Empirical Study
• A mean-field limit for certain deep neural networks
• Mean Field Limit of the Learning Dynamics of Multilayer Neural Networks
• Exact expressions for double descent and implicit regularization via surrogate random design
• Learning through atypical "phase transitions" in overparameterized neural networks
• Triple descent and the two kinds of overfitting: Where & why do they appear?
• On the training dynamics of deep networks with $L_2$ regularization
• Contrasting random and learned features in deep Bayesian linear regression
• Geometric compression of invariant manifolds in neural nets
• Resource Saving via Ensemble Techniques for Quantum Neural Networks
• A Model of Double Descent for High-dimensional Binary Linear Classification
• Scaling Laws for Neural Machine Translation
• What causes the test error? Going beyond bias-variance via ANOVA
• On Power Laws in Deep Ensembles
• Repulsive Deep Ensembles are Bayesian
• Anytime Inference with Distilled Hierarchical Neural Ensembles
• Asymptotics of Wide Convolutional Neural Networks
• Benign Overfitting in Multiclass Classification: All Roads Lead to Interpolation
• Force balance controls the relaxation time of the gradient descent algorithm in the satisfiable phase
• A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learning
• Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra
• Implicit Regularization of Random Feature Models
• Spatially heterogeneous learning by a deep student machine
• On the interplay between data structure and loss function in classification problems
• Finding the Needle in the Haystack with Convolutions: on the benefits of architectural bias
• The asymptotic spectrum of the Hessian of DNN throughout training
• The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspective
• Non-asymptotic approximations of neural networks by Gaussian processes
• Implicit Gradient Regularization
• Perspective: A Phase Diagram for Deep Learning unifying Jamming, Feature Learning and Lazy Training
• A Precise Performance Analysis of Support Vector Regression
• Wider Networks Learn Better Features
• Avoiding The Double Descent Phenomenon of Random Feature Models Using Hybrid Regularization
• Ridgeless Regression with Random Features
• Analyzing Finite Neural Networks: Can We Trust Neural Tangent Kernel Theory?
• Optimization Variance: Exploring Generalization Properties of DNNs
• Deep Learning Classification of Lake Zooplankton
• An analytic theory of shallow networks dynamics for hinge loss classification
• Model, sample, and epoch-wise descents: exact solution of gradient flow in the random feature model
• Rethink the Connections among Generalization, Memorization and the Spectral Bias of DNNs
• On the Bias-Variance Tradeoff: Textbooks Need an Update
• Subspace Fitting Meets Regression: The Effects of Supervision and Orthonormality Constraints on Double Descent of Generalization Errors
• Phases of learning dynamics in artificial neural networks: with or without mislabeled data
• A Convergence Theory Towards Practical Over-parameterized Deep Neural Networks
• Double descent in quantum kernel methods
• The Double Descent Behavior in Two Layer Neural Network for Binary Classification
• Predicting the outputs of finite deep neural networks trained with noisy gradients
• Double Descent Risk and Volume Saturation Effects: A Geometric Perspective
• What needles do sparse neural networks find in nonlinear haystacks
• Normalization effects on shallow neural networks and related asymptotic expansions
• Unintended Effects on Adaptive Learning Rate for Training Neural Network with Output Scale Change
• How isotropic kernels perform on simple invariants
• Large-Dimensional Random Matrix Theory and Its Applications in Deep Learning and Wireless Communications