Early-stopped neural networks are consistent

Abstract

This work studies the behavior of shallow ReLU networks trained with the logistic loss via gradient descent on binary classification data where the underlying data distribution is general, and the (optimal) Bayes risk is not necessarily zero. In this setting, it is shown that gradient descent with early stopping achieves population risk arbitrarily close to optimal in terms of not just logistic and misclassification losses, but also in terms of calibration, meaning the sigmoid mapping of its outputs approximates the true underlying conditional distribution arbitrarily finely. Moreover, the necessary iteration, sample, and architectural complexities of this analysis all scale naturally with a certain complexity measure of the true conditional model. Lastly, while it is not shown that early stopping is necessary, it is shown that any univariate classifier satisfying a local interpolation property is inconsistent.

References in corpus (8)

• Distilling the Knowledge in a Neural Network
• On Calibration of Modern Neural Networks
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
• Towards moderate overparameterization: global convergence guarantees for training shallow neural networks
• Agnostic Learning of a Single Neuron with Gradient Descent
• Parameter-free Stochastic Optimization of Variationally Coherent Functions
• Achieving Small Test Error in Mildly Overparameterized Neural Networks

Cited by in corpus (1)

• Explaining generalization in deep learning: progress and fundamental limits