Memorizing without overfitting: Bias, variance, and interpolation in over-parameterized models

Abstract

The bias-variance trade-off is a central concept in supervised learning. In classical statistics, increasing the complexity of a model (e.g., number of parameters) reduces bias but also increases variance. Until recently, it was commonly believed that optimal performance is achieved at intermediate model complexities which strike a balance between bias and variance. Modern Deep Learning methods flout this dogma, achieving state-of-the-art performance using "over-parameterized models" where the number of fit parameters is large enough to perfectly fit the training data. As a result, understanding bias and variance in over-parameterized models has emerged as a fundamental problem in machine learning. Here, we use methods from statistical physics to derive analytic expressions for bias and variance in two minimal models of over-parameterization (linear regression and two-layer neural networks with nonlinear data distributions), allowing us to disentangle properties stemming from the model architecture and random sampling of data. In both models, increasing the number of fit parameters leads to a phase transition where the training error goes to zero and the test error diverges as a result of the variance (while the bias remains finite). Beyond this threshold, the test error of the two-layer neural network decreases due to a monotonic decrease in \emph{both} the bias and variance in contrast with the classical bias-variance trade-off. We also show that in contrast with classical intuition, over-parameterized models can overfit even in the absence of noise and exhibit bias even if the student and teacher models match. We synthesize these results to construct a holistic understanding of generalization error and the bias-variance trade-off in over-parameterized models and relate our results to random matrix theory.

arXiv comment21 pages (double column), 6 figures, 32 pages of supplemental material (single column)

References in corpus (40)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Reconciling modern machine learning practice and the bias-variance trade-off
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• An Analysis of Deep Neural Network Models for Practical Applications
• A high-bias, low-variance introduction to Machine Learning for physicists
• Benign Overfitting in Linear Regression
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent
• Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Models
• Scaling description of generalization with number of parameters in deep learning
• Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
• Two models of double descent for weak features
• Universality of the SAT-UNSAT (jamming) threshold in non-convex continuous constraint satisfaction problems
• The jamming transition as a paradigm to understand the loss landscape of deep neural networks
• Modelling the influence of data structure on learning in neural networks: the hidden manifold model
• Mean-field message-passing equations in the Hopfield model and its generalizations
• Just Interpolate: Kernel "Ridgeless" Regression Can Generalize
• Rethinking Bias-Variance Trade-off for Generalization of Neural Networks
• A jamming transition from under- to over-parametrization affects loss landscape and generalization
• Surprises in High-Dimensional Ridgeless Least Squares Interpolation
• Disentangling feature and lazy training in deep neural networks
• An analytic theory of generalization dynamics and transfer learning in deep linear networks
• The cavity method at zero temperature
• More Data Can Hurt for Linear Regression: Sample-wise Double Descent
• Understanding overfitting peaks in generalization error: Analytical risk curves for $l_2$ and $l_1$ penalized interpolation
• Exact expressions for double descent and implicit regularization via surrogate random design
• Triple descent and the two kinds of overfitting: Where & why do they appear?
• A New Look at an Old Problem: A Universal Learning Approach to Linear Regression
• Entropic gradient descent algorithms and wide flat minima
• Constrained optimization as ecological dynamics with applications to random quadratic programming in high dimensions
• On the number of variables to use in principal component regression
• Understanding Double Descent Requires a Fine-Grained Bias-Variance Decomposition
• Exact mean field inference in asymmetric kinetic Ising systems
• What causes the test error? Going beyond bias-variance via ANOVA
• Wide flat minima and optimal generalization in classifying high-dimensional Gaussian mixtures
• The Perturbative Resolvent Method: spectral densities of random matrix ensembles via perturbation theory
• Benign Overfitting and Noisy Features
• The cavity method for analysis of large-scale penalized regression
• Critical Behavior and Universality Classes for an Algorithmic Phase Transition in Sparse Reconstruction
• The Geometry of Over-parameterized Regression and Adversarial Perturbations

Cited by in corpus (14)

• Learning without neurons in physical systems
• Learning through atypical "phase transitions" in overparameterized neural networks
• The Physical Effects of Learning
• Random Features for Kernel Approximation: A Survey on Algorithms, Theory, and Beyond
• What causes the test error? Going beyond bias-variance via ANOVA
• On the Double Descent of Random Features Models Trained with SGD
• Supervised Learning Guarantee for Quantum AdaBoost
• Weight fluctuations in (deep) linear neural networks and a derivation of the inverse-variance flatness relation
• The Geometry of Over-parameterized Regression and Adversarial Perturbations
• Simplified derivations for high-dimensional convex learning problems
• Double descent in quantum kernel methods
• Backdoor attacks on DNN and GBDT -- A Case Study from the insurance domain
• Tradeoff of generalization error in unsupervised learning
• Bias-variance decomposition of overparameterized regression with random linear features