Optimal Regularization Can Mitigate Double Descent

Abstract

Recent empirical and theoretical studies have shown that many learning algorithms -- from linear regression to neural networks -- can have test performance that is non-monotonic in quantities such the sample size and model size. This striking phenomenon, often referred to as "double descent", has raised questions of if we need to re-think our current understanding of generalization. In this work, we study whether the double-descent phenomenon can be avoided by using optimal regularization. Theoretically, we prove that for certain linear regression models with isotropic data distribution, optimally-tuned ${\ell }_2$ regularization achieves monotonic test performance as we grow either the sample size or the model size. We also demonstrate empirically that optimally-tuned ${\ell }_2$ regularization can mitigate double descent for more general models, including neural networks. Our results suggest that it may also be informative to study the test risk scalings of various algorithms in the context of appropriately tuned regularization.

arXiv commentv2: Accepted to ICLR 2021. Minor edits to Intro and Appendix

References in corpus (17)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Benign Overfitting in Linear Regression
• The generalization error of random features regression: Precise asymptotics and double descent curve
• Scaling description of generalization with number of parameters in deep learning
• The jamming transition as a paradigm to understand the loss landscape of deep neural networks
• A Modern Take on the Bias-Variance Tradeoff in Neural Networks
• Data-dependent Sample Complexity of Deep Neural Networks via Lipschitz Augmentation
• An analytic theory of generalization dynamics and transfer learning in deep linear networks
• More Data Can Hurt for Linear Regression: Sample-wise Double Descent
• Understanding overfitting peaks in generalization error: Analytical risk curves for $l_2$ and $l_1$ penalized interpolation
• Exact expressions for double descent and implicit regularization via surrogate random design
• Triple descent and the two kinds of overfitting: Where & why do they appear?
• A New Look at an Old Problem: A Universal Learning Approach to Linear Regression
• Improved Sample Complexities for Deep Networks and Robust Classification via an All-Layer Margin
• Ramanujan Graphs and the Solution of the Kadison-Singer Problem
• On the number of variables to use in principal component regression
• Optimal ridge penalty for real-world high-dimensional data can be zero or negative due to the implicit ridge regularization

Cited by in corpus (29)

• Bayesian Deep Learning and a Probabilistic Perspective of Generalization
• Multiple Descent: Design Your Own Generalization Curve
• Triple descent and the two kinds of overfitting: Where & why do they appear?
• A Precise Performance Analysis of Learning with Random Features
• Asymptotics of Ridge (less) Regression under General Source Condition
• Shape Matters: Understanding the Implicit Bias of the Noise Covariance
• Kernel regression in high dimensions: Refined analysis beyond double descent
• Benign Overfitting of Constant-Stepsize SGD for Linear Regression
• Towards an Understanding of Benign Overfitting in Neural Networks
• A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learning
• Dimensionality reduction, regularization, and generalization in overparameterized regressions
• Linear Regression with Distributed Learning: A Generalization Error Perspective
• Early Stopping in Deep Networks: Double Descent and How to Eliminate it
• On the interplay between data structure and loss function in classification problems
• Mehler's Formula, Branching Process, and Compositional Kernels of Deep Neural Networks
• How more data can hurt: Instability and regularization in next-generation reservoir computing
• On the Universality of the Double Descent Peak in Ridgeless Regression
• A Precise Performance Analysis of Support Vector Regression
• Minimum ${\ell }_1$-norm interpolators: Precise asymptotics and multiple descent
• On the connection between least squares, regularization, and classical shadows
• Double Descent and Other Interpolation Phenomena in GANs
• Pre-interpolation loss behaviour in neural networks
• Out-of-Distribution Generalization in Kernel Regression
• Risk-Monotonicity in Statistical Learning
• Revisiting minimum description length complexity in overparameterized models
• Double Descent Risk and Volume Saturation Effects: A Geometric Perspective
• A Statistician Teaches Deep Learning
• Dimension Independent Generalization Error by Stochastic Gradient Descent
• Distribution Free Uncertainty for the Minimum Norm Solution of Over-parameterized Linear Regression