On the Bias-Variance Tradeoff: Textbooks Need an Update

Abstract

The main goal of this thesis is to point out that the bias-variance tradeoff is not always true (e.g. in neural networks). We advocate for this lack of universality to be acknowledged in textbooks and taught in introductory courses that cover the tradeoff. We first review the history of the bias-variance tradeoff, its prevalence in textbooks, and some of the main claims made about the bias-variance tradeoff. Through extensive experiments and analysis, we show a lack of a bias-variance tradeoff in neural networks when increasing network width. Our findings seem to contradict the claims of the landmark work by Geman et al. (1992). Motivated by this contradiction, we revisit the experimental measurements in Geman et al. (1992). We discuss that there was never strong evidence for a tradeoff in neural networks when varying the number of parameters. We observe a similar phenomenon beyond supervised learning, with a set of deep reinforcement learning experiments. We argue that textbook and lecture revisions are in order to convey this nuanced modern understanding of the bias-variance tradeoff.

arXiv commentMSc Thesis

References in corpus (13)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Reconciling modern machine learning practice and the bias-variance trade-off
• An Analysis of Deep Neural Network Models for Practical Applications
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Three Factors Influencing Minima in SGD
• Scaling description of generalization with number of parameters in deep learning
• Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
• The jamming transition as a paradigm to understand the loss landscape of deep neural networks
• Gradient Descent Happens in a Tiny Subspace
• The Shattered Gradients Problem: If resnets are the answer, then what is the question?
• Implicit Regularization in Deep Learning
• Theory of Deep Learning IIb: Optimization Properties of SGD
• The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Study