Learning with Bad Training Data via Iterative Trimmed Loss Minimization

Abstract

In this paper, we study a simple and generic framework to tackle the problem of learning model parameters when a fraction of the training samples are corrupted. We first make a simple observation: in a variety of such settings, the evolution of training accuracy (as a function of training epochs) is different for clean and bad samples. Based on this we propose to iteratively minimize the trimmed loss, by alternating between (a) selecting samples with lowest current loss, and (b) retraining a model on only these samples. We prove that this process recovers the ground truth (with linear convergence rate) in generalized linear models with standard statistical assumptions. Experimentally, we demonstrate its effectiveness in three settings: (a) deep image classifiers with errors only in labels, (b) generative adversarial networks with bad training images, and (c) deep image classifiers with adversarial (image, label) pairs (i.e., backdoor attacks). For the well-studied setting of random label noise, our algorithm achieves state-of-the-art performance without having access to any a-priori guaranteed clean samples.

Cited by in corpus (40)

• DivideMix: Learning with Noisy Labels as Semi-supervised Learning
• Can You Really Backdoor Federated Learning?
• Learning Robust Recommenders through Cross-Model Agreement
• ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reduction
• Identifying Mislabeled Data using the Area Under the Margin Ranking
• Data Valuation using Reinforcement Learning
• Confidence Scores Make Instance-dependent Label-noise Learning Possible
• Learning with Feature-Dependent Label Noise: A Progressive Approach
• Learning from Rules Generalizing Labeled Exemplars
• Meta Transition Adaptation for Robust Deep Learning with Noisy Labels
• Practical Detection of Trojan Neural Networks: Data-Limited and Data-Free Cases
• Certified Robustness to Label-Flipping Attacks via Randomized Smoothing
• How does Early Stopping Help Generalization against Label Noise?
• Towards Understanding Deep Learning from Noisy Labels with Small-Loss Criterion
• Outlier-Robust Optimal Transport
• How benign is benign overfitting?
• Choosing the Sample with Lowest Loss makes SGD Robust
• Robust Training with Ensemble Consensus
• DORO: Distributional and Outlier Robust Optimization
• When Do Curricula Work?
• Robust Training in High Dimensions via Block Coordinate Geometric Median Descent
• Sinkhorn Label Allocation: Semi-Supervised Classification via Annealed Self-Training
• Iterative Least Trimmed Squares for Mixed Linear Regression
• The Resistance to Label Noise in K-NN and DNN Depends on its Concentration
• Extreme Multi-label Classification from Aggregated Labels
• Image recognition from raw labels collected without annotators
• DiPSeN: Differentially Private Self-normalizing Neural Networks For Adversarial Robustness in Federated Learning
• No Regret Sample Selection with Noisy Labels
• Outlier-Robust Optimal Transport: Duality, Structure, and Statistical Analysis
• Multiplicative Reweighting for Robust Neural Network Optimization
• Excess Capacity and Backdoor Poisoning
• Lookup or Exploratory: What is Your Search Intent?
• Learning to Rectify for Robust Learning with Noisy Labels
• Mitigating Memorization in Sample Selection for Learning with Noisy Labels
• INN: A Method Identifying Clean-annotated Samples via Consistency Effect in Deep Neural Networks
• Using Under-trained Deep Ensembles to Learn Under Extreme Label Noise
• Sum of Ranked Range Loss for Supervised Learning
• Learning Mixtures of Low-Rank Models
• Improving Generalization of Deep Fault Detection Models in the Presence of Mislabeled Data
• Batch Inverse-Variance Weighting: Deep Heteroscedastic Regression