Deep Learning is Robust to Massive Label Noise

Abstract

Deep neural networks trained on large supervised datasets have led to impressive results in image classification and other tasks. However, well-annotated datasets can be time-consuming and expensive to collect, lending increased interest to larger but noisy datasets that are more easily obtained. In this paper, we show that deep neural networks are capable of generalizing from training data for which true labels are massively outnumbered by incorrect labels. We demonstrate remarkably high test performance after training on corrupted data from MNIST, CIFAR, and ImageNet. For example, on MNIST we obtain test accuracy above 90 percent even after each clean training example has been diluted with 100 randomly-labeled examples. Such behavior holds across multiple patterns of label noise, even when erroneous labels are biased towards confusing classes. We show that training in this regime requires a significant but manageable increase in dataset size that is related to the factor by which correct labels have been diluted. Finally, we provide an analysis of our results that shows how increasing noise decreases the effective batch size.

References in corpus (6)

• ADADELTA: An Adaptive Learning Rate Method
• One weird trick for parallelizing convolutional neural networks
• Training Deep Neural Networks on Noisy Labels with Bootstrapping
• Training Convolutional Networks with Noisy Labels
• No Spurious Local Minima in Nonconvex Low Rank Problems: A Unified Geometric Analysis
• Failures of Gradient-Based Deep Learning

Cited by in corpus (91)

• Automated Gleason Grading of Prostate Biopsies using Deep Learning
• Deep Learning for Chest X-ray Analysis: A Survey
• Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey
• DeepCervix: A Deep Learning-based Framework for the Classification of Cervical Cells Using Hybrid Deep Feature Fusion Techniques
• EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks
• Are Labels Required for Improving Adversarial Robustness?
• Deep learning for detection and segmentation of artefact and disease instances in gastrointestinal endoscopy
• Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks
• Learning to detect chest radiographs containing lung nodules using visual attention networks
• How to Find Your Friendly Neighborhood: Graph Attention Design with Self-Supervision
• Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks
• Self-Adaptive Training: beyond Empirical Risk Minimization
• Modulation spectral features for speech emotion recognition using deep neural networks
• Responsible and Regulatory Conform Machine Learning for Medicine: A Survey of Challenges and Solutions
• ExoMiner: A Highly Accurate and Explainable Deep Learning Classifier that Validates 301 New Exoplanets
• Probabilistic End-to-end Noise Correction for Learning with Noisy Labels
• SELF: Learning to Filter Noisy Labels with Self-Ensembling
• Unsupervised classification to improve the quality of a bird song recording dataset
• Label Noise Types and Their Effects on Deep Learning
• Guided Anisotropic Diffusion and Iterative Learning for Weakly Supervised Change Detection
• Deep Gamblers: Learning to Abstain with Portfolio Theory
• Self-Supervised Similarity Learning for Digital Pathology
• Distillation $\approx$ Early Stopping? Harvesting Dark Knowledge Utilizing Anisotropic Information Retrieval For Overparameterized Neural Network
• Effect of Annotation Errors on Drone Detection with YOLOv3
• Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behavior
• HYDRA: Hypergradient Data Relevance Analysis for Interpreting Deep Neural Networks
• Effectiveness of Distillation Attack and Countermeasure on Neural Network Watermarking
• Turbulence model reduction by deep learning
• Uncertainty Injection: A Deep Learning Method for Robust Optimization
• Mining User Behaviour from Smartphone data: a literature review
• CurriculumNet: Weakly Supervised Learning from Large-Scale Web Images
• Coherent Gradients: An Approach to Understanding Generalization in Gradient Descent-based Optimization
• Weakly Supervised Change Detection Using Guided Anisotropic Difusion
• MIST: Multiple Instance Self-Training Framework for Video Anomaly Detection
• Simple and Effective Regularization Methods for Training on Noisily Labeled Data with Generalization Guarantee
• Luck Matters: Understanding Training Dynamics of Deep ReLU Networks
• Distributional Generalization: A New Kind of Generalization
• On the geometry of generalization and memorization in deep neural networks
• Breast Mass Detection with Faster R-CNN: On the Feasibility of Learning from Noisy Annotations
• Towards Understanding Deep Learning from Noisy Labels with Small-Loss Criterion
• Analysis of constant-Q filterbank based representations for speech emotion recognition
• Choosing the Sample with Lowest Loss makes SGD Robust
• Underwater object detection using Invert Multi-Class Adaboost with deep learning
• Supervised learning of random quantum circuits via scalable neural networks
• SWAT Watershed Model Calibration using Deep Learning
• Semi-Supervised Node Classification by Graph Convolutional Networks and Extracted Side Information
• Derivative Manipulation for General Example Weighting
• 3D for Free: Crossmodal Transfer Learning using HD Maps
• Robust Training with Ensemble Consensus
• SOSELETO: A Unified Approach to Transfer Learning and Training with Noisy Labels
• On The State of Data In Computer Vision: Human Annotations Remain Indispensable for Developing Deep Learning Models
• Self-Adaptive Training: Bridging Supervised and Self-Supervised Learning
• "Is not the truth the truth?": Analyzing the Impact of User Validations for Bus In/Out Detection in Smartphone-based Surveys
• Label-Noise Robust Multi-Domain Image-to-Image Translation
• An Effective Label Noise Model for DNN Text Classification
• Robust and On-the-fly Dataset Denoising for Image Classification
• Which Strategies Matter for Noisy Label Classification? Insight into Loss and Uncertainty
• Human-Expert-Level Brain Tumor Detection Using Deep Learning with Data Distillation and Augmentation
• Learnability with Indirect Supervision Signals
• Learning Invariant Representations of Social Media Users
• Exponentiated Gradient Reweighting for Robust Training Under Label Noise and Beyond
• Robust Deep Graph Based Learning for Binary Classification
• Deep k-NN for Noisy Labels
• Image recognition from raw labels collected without annotators
• Benchmarking Inference Performance of Deep Learning Models on Analog Devices
• RATT: Leveraging Unlabeled Data to Guarantee Generalization
• Super-resolution of Time-series Labels for Bootstrapped Event Detection
• Intentonomy: a Dataset and Study towards Human Intent Understanding
• Pairwise Relations Discriminator for Unsupervised Raven's Progressive Matrices
• Bayesian Neural Networks with Soft Evidence
• Product Image Recognition with Guidance Learning and Noisy Supervision
• Efficient video annotation with visual interpolation and frame selection guidance
• Deep Learning for Earth Image Segmentation based on Imperfect Polyline Labels with Annotation Errors
• Investigation of Training Label Error Impact on RNN-T
• How Bad is Good enough: Noisy annotations for instrument pose estimation
• Improving Training on Noisy Stuctured Labels
• Detecting Spiky Corruption in Markov Decision Processes
• Reducing Geographic Performance Differential for Face Recognition
• RootPainter3D: Interactive-machine-learning enables rapid and accurate contouring for radiotherapy
• Suppressing Mislabeled Data via Grouping and Self-Attention
• Skeptical Deep Learning with Distribution Correction
• AI Learns to Recognize Bengali Handwritten Digits: Bengali.AI Computer Vision Challenge 2018
• Batch Inverse-Variance Weighting: Deep Heteroscedastic Regression
• Investigating naturalistic hand movements by behavior mining in long-term video and neural recordings
• Autoencoders for strategic decision support
• Point-supervised Segmentation of Microscopy Images and Volumes via Objectness Regularization
• Scheduling and Power Control for Wireless Multicast Systems via Deep Reinforcement Learning
• Towards a New Understanding of the Training of Neural Networks with Mislabeled Training Data
• Multi-Label Learning from Single Positive Labels
• Identification of state functions by physically-guided neural networks with physically-meaningful internal layers
• Pastprop-RNN: improved predictions of the future by correcting the past