Early Stopping without a Validation Set

Abstract

Early stopping is a widely used technique to prevent poor generalization performance when training an over-expressive model by means of gradient-based optimization. To find a good point to halt the optimizer, a common practice is to split the dataset into a training and a smaller validation set to obtain an ongoing estimate of the generalization performance. We propose a novel early stopping criterion based on fast-to-compute local statistics of the computed gradients and entirely removes the need for a held-out validation set. Our experiments show that this is a viable approach in the setting of least-squares and logistic regression, as well as neural networks.

16 pages, 10 figures

References in corpus (1)

• Early Stopping is Nonparametric Variational Inference

Cited by in corpus (11)

• AutoML: A Survey of the State-of-the-Art
• Regularization and Optimization strategies in Deep Convolutional Neural Network
• Channel Distillation: Channel-Wise Attention for Knowledge Distillation
• Towards Realistic Practices In Low-Resource Natural Language Processing: The Development Set
• Implementation of Deep Convolutional Neural Network in Multi-class Categorical Image Classification
• Exploiting All Samples in Low-Resource Sentence Classification: Early Stopping and Initialization Parameters
• Channel-Wise Early Stopping without a Validation Set via NNK Polytope Interpolation
• Cockpit: A Practical Debugging Tool for the Training of Deep Neural Networks
• Multi-Feature Semi-Supervised Learning for COVID-19 Diagnosis from Chest X-ray Images
• Isotonic Data Augmentation for Knowledge Distillation
• Skeptical Deep Learning with Distribution Correction