Generalization by Recognizing Confusion

Abstract

A recently-proposed technique called self-adaptive training augments modern neural networks by allowing them to adjust training labels on the fly, to avoid overfitting to samples that may be mislabeled or otherwise non-representative. By combining the self-adaptive objective with mixup, we further improve the accuracy of self-adaptive models for image recognition; the resulting classifier obtains state-of-the-art accuracies on datasets corrupted with label noise. Robustness to label noise implies a lower generalization gap; thus, our approach also leads to improved generalizability. We find evidence that the Rademacher complexity of these algorithms is low, suggesting a new path towards provable generalization for this type of deep learning model. Last, we highlight a novel connection between difficulties accounting for rare classes and robustness under noise, as rare classes are in a sense indistinguishable from label noise. Our code can be found at https://github.com/Tuxianeer/generalizationconfusion.

arXiv comment12 pages, 3 tables, 2 figures

References in corpus (10)

• mixup: Beyond Empirical Risk Minimization
• Wide Residual Networks
• On Calibration of Modern Neural Networks
• A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
• Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization
• Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks
• Symmetric Cross Entropy for Robust Learning with Noisy Labels
• Uniform convergence may be unable to explain generalization in deep learning
• Gradient Descent Maximizes the Margin of Homogeneous Neural Networks
• Self-Adaptive Training: beyond Empirical Risk Minimization