Understanding Double Descent Requires a Fine-Grained Bias-Variance Decomposition

Abstract

Classical learning theory suggests that the optimal generalization performance of a machine learning model should occur at an intermediate model complexity, with simpler models exhibiting high bias and more complex models exhibiting high variance of the predictive function. However, such a simple trade-off does not adequately describe deep learning models that simultaneously attain low bias and variance in the heavily overparameterized regime. A primary obstacle in explaining this behavior is that deep learning algorithms typically involve multiple sources of randomness whose individual contributions are not visible in the total variance. To enable fine-grained analysis, we describe an interpretable, symmetric decomposition of the variance into terms associated with the randomness from sampling, initialization, and the labels. Moreover, we compute the high-dimensional asymptotic behavior of this decomposition for random feature kernel regression, and analyze the strikingly rich phenomenology that arises. We find that the bias decreases monotonically with the network width, but the variance terms exhibit non-monotonic behavior and can diverge at the interpolation boundary, even in the absence of label noise. The divergence is caused by the \emph{interaction} between sampling and initialization and can therefore be eliminated by marginalizing over samples (i.e. bagging) \emph{or} over the initial parameters (i.e. ensemble learning).

arXiv commentPublished as a conference paper in the Proceedings of the Thirty-fourth Conference on Neural Information Processing Systems; 54 pages; 5 figures. arXiv admin note: text overlap with arXiv:2008.06786

Cited by in corpus (15)

• Explaining Neural Scaling Laws
• Memorizing without overfitting: Bias, variance, and interpolation in over-parameterized models
• Random Features for Kernel Approximation: A Survey on Algorithms, Theory, and Beyond
• What causes the test error? Going beyond bias-variance via ANOVA
• A Random Matrix Perspective on Mixtures of Nonlinearities for Deep Learning
• Understanding Generalization in Adversarial Training via the Bias-Variance Decomposition
• Exact Gap between Generalization Error and Uniform Convergence in Random Feature Models
• The Nonlinearity Coefficient - A Practical Guide to Neural Architecture Design
• On the Double Descent of Random Features Models Trained with SGD
• On the Role of Optimization in Double Descent: A Least Squares Study
• The Geometry of Over-parameterized Regression and Adversarial Perturbations
• Towards Understanding Generalization via Decomposing Excess Risk Dynamics
• Knowledge Sheaves: A Sheaf-Theoretic Framework for Knowledge Graph Embedding
• Harmless interpolation in regression and classification with structured features
• Covariate Shift in High-Dimensional Random Feature Regression