Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regime

Abstract

Deep neural networks can achieve remarkable generalization performances while interpolating the training data perfectly. Rather than the U-curve emblematic of the bias-variance trade-off, their test error often follows a "double descent" - a mark of the beneficial role of overparametrization. In this work, we develop a quantitative theory for this phenomenon in the so-called lazy learning regime of neural networks, by considering the problem of learning a high-dimensional function with random features regression. We obtain a precise asymptotic expression for the bias-variance decomposition of the test error, and show that the bias displays a phase transition at the interpolation threshold, beyond which it remains constant. We disentangle the variances stemming from the sampling of the dataset, from the additive noise corrupting the labels, and from the initialization of the weights. Following up on Geiger et al. 2019, we first show that the latter two contributions are the crux of the double descent: they lead to the overfitting peak at the interpolation threshold and to the decay of the test error upon overparametrization. We then quantify how they are suppressed by ensemble averaging the outputs of K independently initialized estimators. When K is sent to infinity, the test error remains constant beyond the interpolation threshold. We further compare the effects of overparametrizing, ensembling and regularizing. Finally, we present numerical experiments on classic deep learning setups to show that our results hold qualitatively in realistic lazy learning scenarios.

arXiv comment29 pages, 12 figures

References in corpus (21)

• Sequence to Sequence Learning with Neural Networks
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Understanding deep learning requires rethinking generalization
• Deep Neural Networks as Gaussian Processes
• Statistical physics of inference: Thresholds and algorithms
• Phase Transitions in the Coloring of Random Graphs
• Divide and Conquer Kernel Ridge Regression: A Distributed Algorithm with Minimax Optimal Rates
• On Lazy Training in Differentiable Programming
• The generalization error of random features regression: Precise asymptotics and double descent curve
• Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
• Scaling description of generalization with number of parameters in deep learning
• A Landscape Analysis of Constraint Satisfaction Problems
• Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limit
• Generalisation error in learning with random features and the hidden manifold model
• Geometry of Optimization and Implicit Regularization in Deep Learning
• Rethinking Bias-Variance Trade-off for Generalization of Neural Networks
• A Modern Take on the Bias-Variance Tradeoff in Neural Networks
• Level statistics and localization transitions of Lévy matrices
• Kernel and Rich Regimes in Overparametrized Models
• More Data Can Hurt for Linear Regression: Sample-wise Double Descent
• Implicit Regularization of Random Feature Models

Cited by in corpus (16)

• Triple descent and the two kinds of overfitting: Where & why do they appear?
• Geometric compression of invariant manifolds in neural nets
• Probing transfer learning with a model of synthetic correlated datasets
• Random Features for Kernel Approximation: A Survey on Algorithms, Theory, and Beyond
• On Power Laws in Deep Ensembles
• On the Optimal Weighted ${\ell }_2$ Regularization in Overparameterized Linear Regression
• Getting Better from Worse: Augmented Bagging and a Cautionary Tale of Variable Importance
• Introduction to the dynamics of disordered systems: equilibrium and gradient descent
• Prediction Errors for Penalized Regressions based on Generalized Approximate Message Passing
• On the Universality of the Double Descent Peak in Ridgeless Regression
• On the Double Descent of Random Features Models Trained with SGD
• Improved generalization by noise enhancement
• Deep Learning Classification of Lake Zooplankton
• Towards a Unified Quadrature Framework for Large-Scale Kernel Machines
• Double Descent Risk and Volume Saturation Effects: A Geometric Perspective
• How isotropic kernels perform on simple invariants