CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation

Abstract

Annotated data plays a critical role in Natural Language Processing (NLP) in training models and evaluating their performance. Given recent developments in Large Language Models (LLMs), models such as ChatGPT demonstrate zero-shot capability on many text-annotation tasks, comparable with or even exceeding human annotators. Such LLMs can serve as alternatives for manual annotation, due to lower costs and higher scalability. However, limited work has leveraged LLMs as complementary annotators, nor explored how annotation work is best allocated among humans and LLMs to achieve both quality and cost objectives. We propose CoAnnotating, a novel paradigm for Human-LLM co-annotation of unstructured texts at scale. Under this framework, we utilize uncertainty to estimate LLMs' annotation capability. Our empirical study shows CoAnnotating to be an effective means to allocate work from results on different datasets, with up to 21% performance improvement over random baseline. For code implementation, see https://github.com/SALT-NLP/CoAnnotating.

References in corpus (23)

• Character-level Convolutional Networks for Text Classification
• ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks
• Self-Consistency Improves Chain of Thought Reasoning in Language Models
• ChatGPT: Jack of all trades, master of none
• A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity
• Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech
• Weakly-Supervised Neural Text Classification
• Language Models (Mostly) Know What They Know
• ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning
• Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERT
• Large Language Models Are State-of-the-Art Evaluators of Translation Quality
• Yara Parser: A Fast and Accurate Dependency Parser
• ChatGPT: Beginning of an End of Manual Linguistic Data Annotation? Use Case of Automatic Genre Identification
• Teaching Models to Express Their Uncertainty in Words
• Active Prompting with Chain-of-Thought for Large Language Models
• The price of debiasing automatic metrics in natural language evaluation
• A Survey on Programmatic Weak Supervision
• Will Affective Computing Emerge from Foundation Models and General AI? A First Evaluation on ChatGPT
• Snorkel DryBell: A Case Study in Deploying Weak Supervision at Industrial Scale
• Can ChatGPT Replace Traditional KBQA Models? An In-depth Analysis of the Question Answering Performance of the GPT LLM Family
• TempoWiC: An Evaluation Benchmark for Detecting Meaning Shift in Social Media
• SynthBio: A Case Study in Human-AI Collaborative Curation of Text Datasets
• Fast Rates in Pool-Based Batch Active Learning

Cited by in corpus (8)

• Emergent social conventions and collective bias in LLM populations
• Survey and Experiments on Mental Disorder Detection via Social Media: From Large Language Models and RAG to Agents
• Human Evaluation of Procedural Knowledge Graph Extraction from Text with Large Language Models
• CitiLink-Minutes: A Multilayer Annotated Dataset of Municipal Meeting Minutes
• Kastor: Fine-tuned Small Language Models for Shape-based Active Relation Extraction
• ViMRHP: A Vietnamese Benchmark Dataset for Multimodal Review Helpfulness Prediction via Human-AI Collaborative Annotation
• Reliable Annotations with Less Effort: Evaluating LLM-Human Collaboration in Search Clarifications
• Repurposing Annotation Guidelines to Instruct LLM Annotators: A Case Study