Multiple Descent: Design Your Own Generalization Curve

Abstract

This paper explores the generalization loss of linear regression in variably parameterized families of models, both under-parameterized and over-parameterized. We show that the generalization curve can have an arbitrary number of peaks, and moreover, locations of those peaks can be explicitly controlled. Our results highlight the fact that both classical U-shaped generalization curve and the recently observed double descent curve are not intrinsic properties of the model family. Instead, their emergence is due to the interaction between the properties of the data and the inductive biases of learning algorithms.

Accepted to NeurIPS 2021

References in corpus (3)

• Benign overfitting in ridge regression
• Convergence and Alignment of Gradient Descent with Random Backpropagation Weights
• Minimum ${\ell }_1$-norm interpolators: Precise asymptotics and multiple descent

Cited by in corpus (14)

• The Modern Mathematics of Deep Learning
• What causes the test error? Going beyond bias-variance via ANOVA
• Towards an Understanding of Benign Overfitting in Neural Networks
• Taxonomizing local versus global structure in neural network loss landscapes
• Extrapolated cross-validation for randomized ensembles
• On the interplay between data structure and loss function in classification problems
• On the Double Descent of Random Features Models Trained with SGD
• On the Universality of the Double Descent Peak in Ridgeless Regression
• Minimum ${\ell }_1$-norm interpolators: Precise asymptotics and multiple descent
• Generalization error of minimum weighted norm and kernel interpolation
• Out-of-Distribution Generalization in Kernel Regression
• Risk-Monotonicity in Statistical Learning
• Dropout Drops Double Descent
• Mitigating deep double descent by concatenating inputs