Assessing Generalization of SGD via Disagreement

Abstract

We empirically show that the test error of deep networks can be estimated by simply training the same architecture on the same training set but with a different run of Stochastic Gradient Descent (SGD), and measuring the disagreement rate between the two networks on unlabeled test data. This builds on -- and is a stronger version of -- the observation in Nakkiran & Bansal '20, which requires the second run to be on an altogether fresh training set. We further theoretically show that this peculiar phenomenon arises from the \emph{well-calibrated} nature of \emph{ensembles} of SGD-trained models. This finding not only provides a simple empirical measure to directly predict the test error using unlabeled test data, but also establishes a new conceptual connection between generalization and calibration.

References in corpus (22)

• On Calibration of Modern Neural Networks
• Reconciling modern machine learning practice and the bias-variance trade-off
• A Closer Look at Memorization in Deep Networks
• Deep Ensembles: A Loss Landscape Perspective
• A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning
• Measuring Calibration in Deep Learning
• Predicting the Generalization Gap in Deep Networks with Margin Distributions
• Uniform convergence may be unable to explain generalization in deep learning
• Evaluating model calibration in classification
• Generalization in Deep Networks: The Role of Distance from Initialization
• The implicit fairness criterion of unconstrained learning
• Calibration tests in multi-class classification: A unifying framework
• NeurIPS 2020 Competition: Predicting Generalization in Deep Learning
• Don't Just Blame Over-parametrization for Over-confidence: Theoretical Analysis of Calibration in Binary Classification
• Representation Based Complexity Measures for Predicting Generalization in Deep Learning
• Distributional Generalization: A New Kind of Generalization
• Moment Multicalibration for Uncertainty Estimation
• Early-stopped neural networks are consistent
• Should Ensemble Members Be Calibrated?
• RATT: Leveraging Unlabeled Data to Guarantee Generalization

Cited by in corpus (7)

• Understanding Deep Learning via Decision Boundary
• A Framework for Cluster and Classifier Evaluation in the Absence of Reference Labels
• Detecting Errors and Estimating Accuracy on Unlabeled Data with Self-training Ensembles
• Evaluating Robustness to Dataset Shift via Parametric Robustness Sets
• On Predicting Generalization using GANs
• Explaining generalization in deep learning: progress and fundamental limits
• Toward Auto-evaluation with Confidence-based Category Relation-aware Regression