Active label cleaning for improved dataset quality under resource constraints
arXiv:2109.00574 · doi:10.1038/s41467-022-28818-3
Abstract
Imperfections in data annotation, known as label noise, are detrimental to the training of machine learning models and have an often-overlooked confounding effect on the assessment of model performance. Nevertheless, employing experts to remove label noise by fully re-annotating large datasets is infeasible in resource-constrained settings, such as healthcare. This work advocates for a data-driven approach to prioritising samples for re-annotation - which we term "active label cleaning". We propose to rank instances according to estimated label correctness and labelling difficulty of each sample, and introduce a simulation framework to evaluate relabelling efficacy. Our experiments on natural images and on a new medical imaging benchmark show that cleaning noisy labels mitigates their negative impact on model training, evaluation, and selection. Crucially, the proposed active label cleaning enables correcting labels up to 4 times more effectively than typical random selection in realistic conditions, making better use of experts' valuable time for improving dataset quality.
arXiv commentAccepted for publication in Nature Communications
References in corpus (21)
- A Simple Framework for Contrastive Learning of Visual Representations
- ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases
- Bootstrap your own latent: A new approach to self-supervised Learning
- On Calibration of Modern Neural Networks
- Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels
- When Does Label Smoothing Help?
- Active Learning for Convolutional Neural Networks: A Core-Set Approach
- Deep Bayesian Active Learning with Image Data
- Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy
- DivideMix: Learning with Noisy Labels as Semi-supervised Learning
- Bayesian Active Learning for Classification and Preference Learning
- Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels
- A Closer Look at Memorization in Deep Networks
- Early-Learning Regularization Prevents Memorization of Noisy Labels
- Part-dependent Label Noise: Towards Instance-dependent Label Noise
- Learning From Noisy Singly-labeled Data
- An Empirical Study of Example Forgetting during Deep Neural Network Learning
- Learning from Noisy Labels with Deep Neural Networks: A Survey
- Learning explanations that are hard to vary
- Capturing human categorization of natural images at scale by combining deep networks and cognitive models
- SELF: Learning to Filter Noisy Labels with Self-Ensembling