A General Model for Aggregating Annotations Across Simple, Complex, and Multi-Object Annotation Tasks

Abstract

Human annotations are vital to supervised learning, yet annotators often disagree on the correct label, especially as annotation tasks increase in complexity. A strategy to improve label quality is to ask multiple annotators to label the same item and aggregate their labels. Many aggregation models have been proposed for categorical or numerical annotation tasks, but far less work has considered more complex annotation tasks involving open-ended, multivariate, or structured responses. While a variety of bespoke models have been proposed for specific tasks, our work is the first to introduce aggregation methods that generalize across many diverse complex tasks, including sequence labeling, translation, syntactic parsing, ranking, bounding boxes, and keypoints. This generality is achieved by devising a task-agnostic method to model distances between labels rather than the labels themselves. This article extends our prior work with investigation of three new research questions. First, how do complex annotation properties impact aggregation accuracy? Second, how should a task owner navigate the many modeling choices to maximize aggregation accuracy? Finally, what diagnoses can verify that aggregation models are specified correctly for the given data? To understand how various factors impact accuracy and to inform model selection, we conduct simulation studies and experiments on real, complex datasets. Regarding testing, we introduce unit tests for aggregation models and present a suite of such tests to ensure that a given model is not mis-specified and exhibits expected behavior. Beyond investigating these research questions above, we discuss the foundational concept of annotation complexity, present a new aggregation model as a bridge between traditional models and our own, and contribute a new semi-supervised learning method for complex label aggregation that outperforms prior work.

References in corpus (18)

• Efficient Estimation of Word Representations in Vector Space
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• NLTK: The Natural Language Toolkit
• Introduction to the CoNLL-2002 Shared Task: Language-Independent Named Entity Recognition
• A Neural Representation of Sketch Drawings
• Validating Bayesian Inference Algorithms with Simulation-Based Calibration
• Jury Learning: Integrating Dissenting Voices into Machine Learning Models
• Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks
• TGIF: A New Dataset and Benchmark on Animated GIF Description
• Direct Uncertainty Prediction for Medical Second Opinions
• Crowdsourcing Control: Moving Beyond Multiple Choice
• HateCheck: Functional Tests for Hate Speech Detection Models
• TrueLabel + Confusions: A Spectrum of Probabilistic Models in Analyzing Multiple Ratings
• Inferring ground truth from multi-annotator ordinal data: a probabilistic approach
• Metrology for AI: From Benchmarks to Instruments
• Cicero: Multi-Turn, Contextual Argumentation for Accurate Crowdsourcing
• GPM: A Generic Probabilistic Model to Recover Annotator's Behavior and Ground Truth Labeling
• The Challenge of Variable Effort Crowdsourcing and How Visible Gold Can Help