Two models of double descent for weak features

Abstract

The "double descent" risk curve was proposed to qualitatively describe the out-of-sample prediction accuracy of variably-parameterized machine learning models. This article provides a precise mathematical analysis for the shape of this curve in two simple data models with the least squares/least norm predictor. Specifically, it is shown that the risk peaks when the number of features $p$ is close to the sample size $n$, but also that the risk decreases towards its minimum as $p$ increases beyond $n$. This behavior is contrasted with that of "prescient" models that select features in an a priori optimal order.

References in corpus (3)

• Reconciling modern machine learning practice and the bias-variance trade-off
• A jamming transition from under- to over-parametrization affects loss landscape and generalization
• Concentration inequalities for sampling without replacement

Cited by in corpus (16)

• Memorizing without overfitting: Bias, variance, and interpolation in over-parameterized models
• Learning curves of generic features maps for realistic datasets with a teacher-student model
• A Precise High-Dimensional Asymptotic Theory for Boosting and Minimum-${\ell }_1$-Norm Interpolated Classifiers
• Random matrix analysis of deep neural network weight matrices
• Estimation under Model Misspecification with Fake Features
• Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regime
• Overparameterized Linear Regression under Adversarial Attacks
• Dimensionality reduction, regularization, and generalization in overparameterized regressions
• Linear Regression with Distributed Learning: A Generalization Error Perspective
• How more data can hurt: Instability and regularization in next-generation reservoir computing
• New Equivalences Between Interpolation and SVMs: Kernels and Structured Features
• On the connection between least squares, regularization, and classical shadows
• MUSO: Achieving Exact Machine Unlearning in Over-Parameterized Regimes
• The High-Dimensional Asymptotics of Principal Component Regression
• Bias-variance decomposition of overparameterized regression with random linear features
• Dropout Drops Double Descent