Beyond Synthetic Noise: Deep Learning on Controlled Noisy Labels

Abstract

Performing controlled experiments on noisy data is essential in understanding deep learning across noise levels. Due to the lack of suitable datasets, previous research has only examined deep learning on controlled synthetic label noise, and real-world label noise has never been studied in a controlled setting. This paper makes three contributions. First, we establish the first benchmark of controlled real-world label noise from the web. This new benchmark enables us to study the web label noise in a controlled setting for the first time. The second contribution is a simple but effective method to overcome both synthetic and real noisy labels. We show that our method achieves the best result on our dataset as well as on two public benchmarks (CIFAR and WebVision). Third, we conduct the largest study by far into understanding deep neural networks trained on noisy labels across different noise levels, noise types, network architectures, and training settings. The data and code are released at the following link: http://www.lujiang.info/cnlw.html

published at ICML 2020

Cited by in corpus (36)

• LongReMix: Robust Learning with High Confidence Samples in a Noisy Label Environment
• Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks
• A Survey of Label-noise Representation Learning: Past, Present and Future
• Contrast to Divide: Self-Supervised Pre-Training for Learning with Noisy Labels
• Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations
• Sample Selection with Uncertainty of Losses for Learning with Noisy Labels
• WRENCH: A Comprehensive Benchmark for Weak Supervision
• MoPro: Webly Supervised Learning with Momentum Prototypes
• Learning Noise Transition Matrix from Only Noisy Labels via Total Variation Regularization
• Extended T: Learning with Mixed Closed-set and Open-set Noisy Labels
• ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks
• SimAug: Learning Robust Representations from Simulation for Trajectory Prediction
• FedDebug: Systematic Debugging for Federated Learning Applications
• Beyond Class-Conditional Assumption: A Primary Attempt to Combat Instance-Dependent Label Noise
• Towards Understanding Deep Learning from Noisy Labels with Small-Loss Criterion
• A Simple Probabilistic Method for Deep Classification under Input-Dependent Label Noise
• A Second-Order Approach to Learning with Instance-Dependent Label Noise
• Regularizing Generative Adversarial Networks under Limited Data
• Derivative Manipulation for General Example Weighting
• When Do Curricula Work?
• Instance Correction for Learning with Open-set Noisy Labels
• Faster Meta Update Strategy for Noise-Robust Deep Learning
• "Is not the truth the truth?": Analyzing the Impact of User Validations for Bus In/Out Detection in Smartphone-based Surveys
• Deep Learning with Label Differential Privacy
• Beyond Linear Subspace Clustering: A Comparative Study of Nonlinear Manifold Clustering Algorithms
• An Ensemble Noise-Robust K-fold Cross-Validation Selection Method for Noisy Labels
• Image recognition from raw labels collected without annotators
• Learning from Multiple Annotators by Incorporating Instance Features
• Reweighting Augmented Samples by Minimizing the Maximal Expected Loss
• Noise-Resistant Deep Metric Learning with Probabilistic Instance Filtering
• From Recognition to Prediction: Analysis of Human Action and Trajectory Prediction in Video
• ARCA23K: An audio dataset for investigating open-set label noise
• Detecting Label Noise via Leave-One-Out Cross-Validation
• Combating Label Noise With A General Surrogate Model For Sample Selection
• INN: A Method Identifying Clean-annotated Samples via Consistency Effect in Deep Neural Networks
• Learning advisor networks for noisy image classification