Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks

Abstract

We identify label errors in the test sets of 10 of the most commonly-used computer vision, natural language, and audio datasets, and subsequently study the potential for these label errors to affect benchmark results. Errors in test sets are numerous and widespread: we estimate an average of at least 3.3% errors across the 10 datasets, where for example label errors comprise at least 6% of the ImageNet validation set. Putative label errors are identified using confident learning algorithms and then human-validated via crowdsourcing (51% of the algorithmically-flagged candidates are indeed erroneously labeled, on average across the datasets). Traditionally, machine learning practitioners choose which model to deploy based on test accuracy - our findings advise caution here, proposing that judging models over correctly labeled test sets may be more useful, especially for noisy real-world datasets. Surprisingly, we find that lower capacity models may be practically more useful than higher capacity models in real-world datasets with high proportions of erroneously labeled data. For example, on ImageNet with corrected labels: ResNet-18 outperforms ResNet-50 if the prevalence of originally mislabeled test examples increases by just 6%. On CIFAR-10 with corrected labels: VGG-11 outperforms VGG-19 if the prevalence of originally mislabeled test examples increases by just 5%. Test set errors across the 10 datasets can be viewed at https://labelerrors.com and all label errors can be reproduced by https://github.com/cleanlab/label-errors.

arXiv commentDemo available at https://labelerrors.com/ and source code available at https://github.com/cleanlab/label-errors

References in corpus (11)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• A Neural Representation of Sketch Drawings
• Deep Learning is Robust to Massive Label Noise
• Do ImageNet Classifiers Generalize to ImageNet?
• A Closer Look at Memorization in Deep Networks
• WebVision Database: Visual Learning and Understanding from Web Data
• Beyond Synthetic Noise: Deep Learning on Controlled Noisy Labels
• Understanding and Utilizing Deep Neural Networks Trained with Noisy Labels
• What Do Compressed Deep Neural Networks Forget?
• From ImageNet to Image Classification: Contextualizing Progress on Benchmarks
• Understanding Generalization through Visualizations

Cited by in corpus (18)

• Partial success in closing the gap between human and machine vision
• Online Deterministic Annealing for Classification and Clustering
• FOD-A: A Dataset for Foreign Object Debris in Airports
• The Benchmark Lottery
• Data Curation and Quality Assurance for Machine Learning-based Cyber Intrusion Detection
• Data Quality Matters For Adversarial Training: An Empirical Study
• A General Model for Aggregating Annotations Across Simple, Complex, and Multi-Object Annotation Tasks
• When and how epochwise double descent happens
• The Influence of Faulty Labels in Data Sets on Human Pose Estimation
• Domain Adaptation on Semantic Segmentation with Separate Affine Transformation in Batch Normalization
• Cleaning and Structuring the Label Space of the iMet Collection 2020
• MixDefense: A Defense-in-Depth Framework for Adversarial Example Detection Based on Statistical and Semantic Analysis
• Rethinking Noisy Label Models: Labeler-Dependent Noise with Adversarial Awareness
• Trivial or impossible -- dichotomous data difficulty masks model differences (on ImageNet and beyond)
• A Tale Of Two Long Tails
• Sample Noise Impact on Active Learning
• Extracting and filtering paraphrases by bridging natural language inference and paraphrasing
• Reformulating van Rijsbergen's $F_{\beta }$ metric for weighted binary cross-entropy