The Geometry of Over-parameterized Regression and Adversarial Perturbations

Abstract

Classical regression has a simple geometric description in terms of a projection of the training labels onto the column space of the design matrix. However, for over-parameterized models -- where the number of fit parameters is large enough to perfectly fit the training data -- this picture becomes uninformative. Here, we present an alternative geometric interpretation of regression that applies to both under- and over-parameterized models. Unlike the classical picture which takes place in the space of training labels, our new picture resides in the space of input features. This new feature-based perspective provides a natural geometric interpretation of the double-descent phenomenon in the context of bias and variance, explaining why it can occur even in the absence of label noise. Furthermore, we show that adversarial perturbations -- small perturbations to the input features that result in large changes in label values -- are a generic feature of biased models, arising from the underlying geometry. We demonstrate these ideas by analyzing three minimal models for over-parameterized linear least squares regression: without basis functions (input features equal model features) and with linear or nonlinear basis functions (two-layer neural networks with linear or nonlinear activation functions, respectively).

arXiv comment11 pages (single column), 4 figures, 10 pages of supporting material

References in corpus (14)

• Reconciling modern machine learning practice and the bias-variance trade-off
• Benign Overfitting in Linear Regression
• The generalization error of random features regression: Precise asymptotics and double descent curve
• Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Models
• Memorizing without overfitting: Bias, variance, and interpolation in over-parameterized models
• An analytic theory of generalization dynamics and transfer learning in deep linear networks
• More Data Can Hurt for Linear Regression: Sample-wise Double Descent
• Understanding overfitting peaks in generalization error: Analytical risk curves for $l_2$ and $l_1$ penalized interpolation
• A New Look at an Old Problem: A Universal Learning Approach to Linear Regression
• Triple descent and the two kinds of overfitting: Where & why do they appear?
• Understanding Double Descent Requires a Fine-Grained Bias-Variance Decomposition
• On the number of variables to use in principal component regression
• What causes the test error? Going beyond bias-variance via ANOVA
• Benign Overfitting and Noisy Features

Cited by in corpus (1)

• Memorizing without overfitting: Bias, variance, and interpolation in over-parameterized models