Multiple Descent: Design Your Own Generalization Curve

Abstract

This paper explores the generalization loss of linear regression in variably parameterized families of models, both under-parameterized and over-parameterized. We show that the generalization curve can have an arbitrary number of peaks, and moreover, locations of those peaks can be explicitly controlled. Our results highlight the fact that both classical U-shaped generalization curve and the recently observed double descent curve are not intrinsic properties of the model family. Instead, their emergence is due to the interaction between the properties of the data and the inductive biases of learning algorithms.

arXiv commentAccepted to NeurIPS 2021

References in corpus (14)

• A Convergence Theory for Deep Learning via Over-Parameterization
• Benign Overfitting in Linear Regression
• On Exact Computation with an Infinitely Wide Neural Net
• The generalization error of random features regression: Precise asymptotics and double descent curve
• Generalization Properties of Learning with Random Features
• Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks
• Benign overfitting in ridge regression
• A Modern Take on the Bias-Variance Tradeoff in Neural Networks
• Optimal Regularization Can Mitigate Double Descent
• Triple descent and the two kinds of overfitting: Where & why do they appear?
• On the number of variables to use in principal component regression
• Asymptotics of Ridge (less) Regression under General Source Condition
• Convergence and Alignment of Gradient Descent with Random Backpropagation Weights
• Minimum ${\ell }_1$-norm interpolators: Precise asymptotics and multiple descent

Cited by in corpus (18)

• The Modern Mathematics of Deep Learning
• Triple descent and the two kinds of overfitting: Where & why do they appear?
• Asymptotics of Ridge (less) Regression under General Source Condition
• Random Features for Kernel Approximation: A Survey on Algorithms, Theory, and Beyond
• What causes the test error? Going beyond bias-variance via ANOVA
• Towards an Understanding of Benign Overfitting in Neural Networks
• Taxonomizing local versus global structure in neural network loss landscapes
• Extrapolated cross-validation for randomized ensembles
• On the interplay between data structure and loss function in classification problems
• Empirical Risk Minimization in the Interpolating Regime with Application to Neural Network Learning
• On the Double Descent of Random Features Models Trained with SGD
• On the Universality of the Double Descent Peak in Ridgeless Regression
• Minimum ${\ell }_1$-norm interpolators: Precise asymptotics and multiple descent
• Risk-Monotonicity in Statistical Learning
• Generalization error of minimum weighted norm and kernel interpolation
• Out-of-Distribution Generalization in Kernel Regression
• Dropout Drops Double Descent
• Mitigating deep double descent by concatenating inputs