Dimension Independent Generalization Error by Stochastic Gradient Descent

Abstract

One classical canon of statistics is that large models are prone to overfitting, and model selection procedures are necessary for high dimensional data. However, many overparameterized models, such as neural networks, perform very well in practice, although they are often trained with simple online methods and regularization. The empirical success of overparameterized models, which is often known as benign overfitting, motivates us to have a new look at the statistical generalization theory for online optimization. In particular, we present a general theory on the generalization error of stochastic gradient descent (SGD) solutions for both convex and locally convex loss functions. We further discuss data and model conditions that lead to a ``low effective dimension". Under these conditions, we show that the generalization error either does not depend on the ambient dimension $p$ or depends on $p$ via a poly-logarithmic factor. We also demonstrate that in several widely used statistical models, the ``low effective dimension'' arises naturally in overparameterized settings. The studied statistical applications include both convex models such as linear regression and logistic regression and non-convex models such as $M$-estimator and two-layer neural networks.

arXiv comment60 pages, 2 figures

References in corpus (13)

• Reconciling modern machine learning practice and the bias-variance trade-off
• Benign Overfitting in Linear Regression
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)
• Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
• Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolate
• Functional additive regression
• A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models Under Gradient Descent Dynamics
• Optimal Regularization Can Mitigate Double Descent
• A Priori Estimates of the Population Risk for Residual Networks
• More Data Can Hurt for Linear Regression: Sample-wise Double Descent
• Learning with SGD and Random Features

Cited by in corpus (1)

• On the Double Descent of Random Features Models Trained with SGD