Early-stopped neural networks are consistent
arXiv:2106.05932
Abstract
This work studies the behavior of shallow ReLU networks trained with the logistic loss via gradient descent on binary classification data where the underlying data distribution is general, and the (optimal) Bayes risk is not necessarily zero. In this setting, it is shown that gradient descent with early stopping achieves population risk arbitrarily close to optimal in terms of not just logistic and misclassification losses, but also in terms of calibration, meaning the sigmoid mapping of its outputs approximates the true underlying conditional distribution arbitrarily finely. Moreover, the necessary iteration, sample, and architectural complexities of this analysis all scale naturally with a certain complexity measure of the true conditional model. Lastly, while it is not shown that early stopping is necessary, it is shown that any univariate classifier satisfying a local interpolation property is inconsistent.
References in corpus (20)
- Distilling the Knowledge in a Neural Network
- On Calibration of Modern Neural Networks
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- A Convergence Theory for Deep Learning via Over-Parameterization
- Benign Overfitting in Linear Regression
- Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
- Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
- Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
- Generalization Bounds of Stochastic Gradient Descent for Wide and Deep Neural Networks
- Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layers
- Towards moderate overparameterization: global convergence guarantees for training shallow neural networks
- Risk and parameter convergence of logistic regression
- Gradient Descent can Learn Less Over-parameterized Two-layer Neural Networks on Classification Problems
- Don't Just Blame Over-parametrization for Over-confidence: Theoretical Analysis of Calibration in Binary Classification
- Agnostic Learning of a Single Neuron with Gradient Descent
- Learning a Single Neuron with Gradient Methods
- Superpolynomial Lower Bounds for Learning One-Layer Neural Networks using Gradient Descent
- Boosting with the Logistic Loss is Consistent
- Parameter-free Stochastic Optimization of Variationally Coherent Functions
- Achieving Small Test Error in Mildly Overparameterized Neural Networks