Effective Degrees of Freedom: A Flawed Metaphor

Abstract

To most applied statisticians, a fitting procedure's degrees of freedom is synonymous with its model complexity, or its capacity for overfitting to data. In particular, it is often used to parameterize the bias-variance tradeoff in model selection. We argue that, contrary to folk intuition, model complexity and degrees of freedom are not synonymous and may correspond very poorly. We exhibit and theoretically explore various examples of fitting procedures for which degrees of freedom is not monotonic in the model complexity parameter, and can exceed the total dimension of the response space. Even in very simple settings, the degrees of freedom can exceed the dimension of the ambient space by an arbitrarily large amount. We show the degrees of freedom for any non-convex projection method can be unbounded.

References in corpus (2)

• On the "degrees of freedom" of the lasso
• When Does More Regularization Imply Fewer Degrees of Freedom? Sufficient Conditions and Counter Examples from Lasso and Ridge Regression

Cited by in corpus (4)

• A State Space Approach for Piecewise-Linear Recurrent Neural Networks for Reconstructing Nonlinear Dynamics from Neural Measurements
• Some models are useful, but how do we know which ones? Towards a unified Bayesian model taxonomy
• P-splines with an l1 penalty for repeated measures
• yaglm: a Python package for fitting and tuning generalized linear models that supports structured, adaptive and non-convex penalties