Learning From Noisy Singly-labeled Data

Abstract

Supervised learning depends on annotated examples, which are taken to be the \emph{ground truth}. But these labels often come from noisy crowdsourcing platforms, like Amazon Mechanical Turk. Practitioners typically collect multiple labels per example and aggregate the results to mitigate noise (the classic crowdsourcing problem). Given a fixed annotation budget and unlimited unlabeled data, redundant annotation comes at the expense of fewer labeled examples. This raises two fundamental questions: (1) How can we best learn from noisy workers? (2) How should we allocate our labeling budget to maximize the performance of a classifier? We propose a new algorithm for jointly modeling labels and worker quality from noisy crowd-sourced data. The alternating minimization proceeds in rounds, estimating worker quality from disagreement with the current model and then updating the model by optimizing a loss function that accounts for the current estimate of worker quality. Unlike previous approaches, even with only one annotation per example, our algorithm can estimate worker quality. We establish a generalization error bound for models learned with our algorithm and establish theoretically that it's better to label many examples once (vs less multiply) when worker quality is above a threshold. Experiments conducted on both ImageNet (with simulated noisy workers) and MS-COCO (using the real crowdsourced labels) confirm our algorithm's benefits.

18 pages, 3 figures

References in corpus (2)

• Training Convolutional Networks with Noisy Labels
• Regularized Minimax Conditional Entropy for Crowdsourcing

Cited by in corpus (40)

• Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels
• Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey
• Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks
• What is the Effect of Importance Weighting in Deep Learning?
• Curriculum Loss: Robust Learning and Generalization against Label Corruption
• Active label cleaning for improved dataset quality under resource constraints
• Ontology-driven weak supervision for clinical entity classification in electronic health records
• Disentangling Human Error from the Ground Truth in Segmentation of Medical Images
• Towards countering hate speech against journalists on social media
• Practical Text Classification With Large Pre-Trained Language Models
• Label Noise Types and Their Effects on Deep Learning
• Confidence Scores Make Instance-dependent Label-noise Learning Possible
• Trustable Co-label Learning from Multiple Noisy Annotators
• Distillation $\approx$ Early Stopping? Harvesting Dark Knowledge Utilizing Anisotropic Information Retrieval For Overparameterized Neural Network
• Using Error Decay Prediction to Overcome Practical Issues of Deep Active Learning for Named Entity Recognition
• Fast and Three-rious: Speeding Up Weak Supervision with Triplet Methods
• Reinforcement Learning with Perturbed Rewards
• Asking Crowdworkers to Write Entailment Examples: The Best of Bad Options
• Training Complex Models with Multi-Task Weak Supervision
• Noisy Batch Active Learning with Deterministic Annealing
• Constrained Labeling for Weakly Supervised Learning
• Multi-Class Classification from Noisy-Similarity-Labeled Data
• Tilted Empirical Risk Minimization
• Learning from Imperfect Annotations
• A Fine-Grained Analysis on Distribution Shift
• Multi-Resolution Weak Supervision for Sequential Data
• VILD: Variational Imitation Learning with Diverse-quality Demonstrations
• Leveraging Just a Few Keywords for Fine-Grained Aspect Detection Through Weakly Supervised Co-Training
• Learning Image Labels On-the-fly for Training Robust Classification Models
• Creating Training Sets via Weak Indirect Supervision
• NLNL: Negative Learning for Noisy Labels
• Deep k-NN for Noisy Labels
• Generative-Discriminative Complementary Learning
• Towards Robust Object Detection: Bayesian RetinaNet for Homoscedastic Aleatoric Uncertainty Modeling
• A Machine Learning Framework Towards Transparency in Experts' Decision Quality
• Learning from Subjective Ratings Using Auto-Decoded Deep Latent Embeddings
• End-to-End Annotator Bias Approximation on Crowdsourced Single-Label Sentiment Analysis
• Cost-Accuracy Aware Adaptive Labeling for Active Learning
• A Non-Intrusive Correction Algorithm for Classification Problems with Corrupted Data
• End-to-End Weak Supervision