Rethinking Bias-Variance Trade-off for Generalization of Neural Networks

Abstract

The classical bias-variance trade-off predicts that bias decreases and variance increase with model complexity, leading to a U-shaped risk curve. Recent work calls this into question for neural networks and other over-parameterized models, for which it is often observed that larger models generalize better. We provide a simple explanation for this by measuring the bias and variance of neural networks: while the bias is monotonically decreasing as in the classical theory, the variance is unimodal or bell-shaped: it increases then decreases with the width of the network. We vary the network architecture, loss function, and choice of dataset and confirm that variance unimodality occurs robustly for all models we considered. The risk curve is the sum of the bias and variance curves and displays different qualitative shapes depending on the relative scale of bias and variance, with the double descent curve observed in recent literature as a special case. We corroborate these empirical results with a theoretical analysis of two-layer linear networks with random first layer. Finally, evaluation on out-of-distribution data shows that most of the drop in accuracy comes from increased bias while variance increases by a relatively small amount. Moreover, we find that deeper models decrease bias and increase variance for both in-distribution and out-of-distribution data.

References in corpus (6)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Reconciling modern machine learning practice and the bias-variance trade-off
• Benign Overfitting in Linear Regression
• The generalization error of random features regression: Precise asymptotics and double descent curve
• More Data Can Hurt for Linear Regression: Sample-wise Double Descent
• From Fixed-X to Random-X Regression: Bias-Variance Decompositions, Covariance Penalties, and Prediction Error Estimation

Cited by in corpus (18)

• An Investigation of Why Overparameterization Exacerbates Spurious Correlations
• Memorizing without overfitting: Bias, variance, and interpolation in over-parameterized models
• Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regime
• Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspective
• Membership Inference Attacks and Generalization: A Causal Perspective
• Understanding Generalization in Adversarial Training via the Bias-Variance Decomposition
• Dimensionality reduction, regularization, and generalization in overparameterized regressions
• Taxonomizing local versus global structure in neural network loss landscapes
• Early Stopping in Deep Networks: Double Descent and How to Eliminate it
• Increasing Depth Leads to U-Shaped Test Risk in Over-parameterized Convolutional Networks
• Recovery and Generalization in Over-Realized Dictionary Learning
• On the connection between least squares, regularization, and classical shadows
• Rethink the Connections among Generalization, Memorization and the Spectral Bias of DNNs
• Comparing the Value of Labeled and Unlabeled Data in Method-of-Moments Latent Variable Estimation
• Optimization Variance: Exploring Generalization Properties of DNNs
• Robustness via Cross-Domain Ensembles
• Vulnerability Under Adversarial Machine Learning: Bias or Variance?
• Mitigating deep double descent by concatenating inputs