RATT: Leveraging Unlabeled Data to Guarantee Generalization
arXiv:2105.00303
Abstract
To assess generalization, machine learning scientists typically either (i) bound the generalization gap and then (after training) plug in the empirical risk to obtain a bound on the true risk; or (ii) validate empirically on holdout data. However, (i) typically yields vacuous guarantees for overparameterized models. Furthermore, (ii) shrinks the training set and its guarantee erodes with each re-use of the holdout set. In this paper, we introduce a method that leverages unlabeled data to produce generalization bounds. After augmenting our (labeled) training set with randomly labeled fresh examples, we train in the standard fashion. Whenever classifiers achieve low error on clean data and high error on noisy data, our bound provides a tight upper bound on the true risk. We prove that our bound is valid for 0-1 empirical risk minimization and with linear classifiers trained by gradient descent. Our approach is especially useful in conjunction with deep learning due to the early learning phenomenon whereby networks fit true labels before noisy labels but requires one intuitive assumption. Empirically, on canonical computer vision and NLP tasks, our bound provides non-vacuous generalization guarantees that track actual performance closely. This work provides practitioners with an option for certifying the generalization of deep nets even when unseen labeled data is unavailable and provides theoretical insights into the relationship between random label noise and generalization.
arXiv commentICML 2021 (Long Talk)
References in corpus (27)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- A Convergence Theory for Deep Learning via Over-Parameterization
- Deep Learning is Robust to Massive Label Noise
- Do ImageNet Classifiers Generalize to ImageNet?
- Gradient Descent Provably Optimizes Over-parameterized Neural Networks
- A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
- Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
- Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
- On Lazy Training in Differentiable Programming
- Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
- Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks
- Uniform convergence may be unable to explain generalization in deep learning
- Gradient Descent Happens in a Tiny Subspace
- Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approach
- Norm-Based Capacity Control in Neural Networks
- Critical Learning Periods in Deep Neural Networks
- The Ladder: A Reliable Leaderboard for Machine Learning Competitions
- Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Tasks
- The Early Phase of Neural Network Training
- Classification vs regression in overparameterized regimes: Does the loss function matter?
- Deterministic PAC-Bayesian generalization bounds for deep networks via generalizing noise-resilience
- The Implicit Regularization of Stochastic Gradient Flow for Least Squares
- Simple and Effective Regularization Methods for Training on Noisily Labeled Data with Generalization Guarantee
- A Continuous-Time View of Early Stopping for Least Squares
- On Uniform Convergence and Low-Norm Interpolation Learning
- An Exponential Efron-Stein Inequality for Lq Stable Learning Rules