Double Descent Risk and Volume Saturation Effects: A Geometric Perspective

Abstract

The appearance of the double-descent risk phenomenon has received growing interest in the machine learning and statistics community, as it challenges well-understood notions behind the U-shaped train-test curves. Motivated through Rissanen's minimum description length (MDL), Balasubramanian's Occam's Razor, and Amari's information geometry, we investigate how the logarithm of the model volume: $\log V$, works to extend intuition behind the AIC and BIC model selection criteria. We find that for the particular model classes of isotropic linear regression and statistical lattices, the $\log V$ term may be decomposed into a sum of distinct components, each of which assist in their explanations of the appearance of this phenomenon. In particular they suggest why generalization error does not necessarily continue to grow with increasing model dimensionality.

arXiv commentUpdated version. Some parts have been re-structured, and certain elements shifted to Appendix

References in corpus (11)

• Reconciling modern machine learning practice and the bias-variance trade-off
• Benign Overfitting in Linear Regression
• Information-theoretic analysis of generalization capability of learning algorithms
• Scaling description of generalization with number of parameters in deep learning
• The jamming transition as a paradigm to understand the loss landscape of deep neural networks
• To understand deep learning we need to understand kernel learning
• Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regime
• Optimal Regularization Can Mitigate Double Descent
• A Geometric Formulation of Occam's Razor for Inference of Parametric Distributions
• Legendre Decomposition for Tensors
• Tensor Balancing on Statistical Manifold