Risk Bounds for Over-parameterized Maximum Margin Classification on Sub-Gaussian Mixtures
arXiv:2104.13628
Abstract
Modern machine learning systems such as deep neural networks are often highly over-parameterized so that they can fit the noisy training data exactly, yet they can still achieve small test errors in practice. In this paper, we study this "benign overfitting" phenomenon of the maximum margin classifier for linear classification problems. Specifically, we consider data generated from sub-Gaussian mixtures, and provide a tight risk bound for the maximum margin linear classifier in the over-parameterized setting. Our results precisely characterize the condition under which benign overfitting can occur in linear classification problems, and improve on previous work. They also have direct implications for over-parameterized logistic regression.
arXiv comment27 pages, 3 figures. In NeurIPS 2021
References in corpus (9)
- Reconciling modern machine learning practice and the bias-variance trade-off
- Benign Overfitting in Linear Regression
- The generalization error of random features regression: Precise asymptotics and double descent curve
- Characterizing Implicit Bias in Terms of Optimization Geometry
- Benign overfitting in ridge regression
- Classification vs regression in overparameterized regimes: Does the loss function matter?
- Stochastic Gradient Descent on Separable Data: Exact Convergence with a Fixed Learning Rate
- A Chernoff-type Lower Bound for the Gaussian Q-function
- On the proliferation of support vectors in high dimensions
Cited by in corpus (8)
- Benign Overfitting in Multiclass Classification: All Roads Lead to Interpolation
- A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learning
- Towards an Understanding of Benign Overfitting in Neural Networks
- Support vector machines and linear regression coincide with very high-dimensional features
- Learning Gaussian Mixtures with Generalised Linear Models: Precise Asymptotics in High-dimensions
- Explaining generalization in deep learning: progress and fundamental limits
- Harmless interpolation in regression and classification with structured features
- Classification and Adversarial examples in an Overparameterized Linear Model: A Signal Processing Perspective