Self-Adaptive Training: beyond Empirical Risk Minimization

Abstract

We propose self-adaptive training---a new training algorithm that dynamically corrects problematic training labels by model predictions without incurring extra computational cost---to improve generalization of deep learning for potentially corrupted training data. This problem is crucial towards robustly learning from data that are corrupted by, e.g., label noises and out-of-distribution samples. The standard empirical risk minimization (ERM) for such data, however, may easily overfit noises and thus suffers from sub-optimal performance. In this paper, we observe that model predictions can substantially benefit the training process: self-adaptive training significantly improves generalization over ERM under various levels of noises, and mitigates the overfitting issue in both natural and adversarial training. We evaluate the error-capacity curve of self-adaptive training: the test error is monotonously decreasing w.r.t. model capacity. This is in sharp contrast to the recently-discovered double-descent phenomenon in ERM which might be a result of overfitting of noises. Experiments on CIFAR and ImageNet datasets verify the effectiveness of our approach in two applications: classification with label noise and selective classification. We release our code at https://github.com/LayneH/self-adaptive-training.

arXiv commentTo appear in NeurIPS 2020

References in corpus (22)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• mixup: Beyond Empirical Risk Minimization
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Wide Residual Networks
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Understanding deep learning requires rethinking generalization
• Theoretically Principled Trade-off between Robustness and Accuracy
• Learning to Reweight Examples for Robust Deep Learning
• Training Deep Neural Networks on Noisy Labels with Bootstrapping
• Deep Learning is Robust to Massive Label Noise
• A Closer Look at Memorization in Deep Networks
• Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks
• Label Refinery: Improving ImageNet Classification through Label Progression
• The jamming transition as a paradigm to understand the loss landscape of deep neural networks
• Uniform convergence may be unable to explain generalization in deep learning
• Combating Label Noise in Deep Learning Using Abstention
• SelectiveNet: A Deep Neural Network with an Integrated Reject Option
• SELF: Learning to Filter Noisy Labels with Self-Ensembling
• Distillation $\approx$ Early Stopping? Harvesting Dark Knowledge Utilizing Anisotropic Information Retrieval For Overparameterized Neural Network

Cited by in corpus (14)

• Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples
• Fixing Data Augmentation to Improve Adversarial Robustness
• Robust Preference-Guided Denoising for Graph based Social Recommendation
• Breaking the Dilemma of Medical Image-to-image Translation
• On the Effects of Different Types of Label Noise in Multi-Label Remote Sensing Image Classification
• Dirichlet-Based Prediction Calibration for Learning with Noisy Labels
• Bag of Tricks for Adversarial Training
• Voting based ensemble improves robustness of defensive models
• Adversarial Robustness through the Lens of Convolutional Filters
• Confidence Adaptive Regularization for Deep Learning with Noisy Labels
• Data Quality Matters For Adversarial Training: An Empirical Study
• Model-Agnostic Meta-Attack: Towards Reliable Evaluation of Adversarial Robustness
• Adversarial Attacks on ML Defense Models Competition
• Generalization by Recognizing Confusion