Reliable Annotations with Less Effort: Evaluating LLM-Human Collaboration in Search Clarifications

Abstract

Despite growing interest in using large language models (LLMs) to automate annotation, their effectiveness in complex, nuanced, and multi-dimensional labelling tasks remains relatively underexplored. This study focuses on annotation for the search clarification task, leveraging a high-quality, multi-dimensional dataset that includes five distinct fine-grained annotation subtasks. Although LLMs have shown impressive capabilities in general settings, our study reveals that even state-of-the-art models struggle to replicate human-level performance in subjective or fine-grained evaluation tasks. Through a systematic assessment, we demonstrate that LLM predictions are often inconsistent, poorly calibrated, and highly sensitive to prompt variations. To address these limitations, we propose a simple yet effective human-in-the-loop (HITL) workflow that uses confidence thresholds and inter-model disagreement to selectively involve human review. Our findings show that this lightweight intervention significantly improves annotation reliability while reducing human effort by up to 45%, offering a relatively scalable and cost-effective yet accurate path forward for deploying LLMs in real-world evaluation settings.

arXiv comment9 pages,5 figures

References in corpus (13)

• ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Is ChatGPT better than Human Annotators? Potential and Limitations of ChatGPT in Explaining Implicit Hate Speech
• Asking Clarifying Questions in Open-Domain Information-Seeking Conversations
• Perspectives on Large Language Models for Relevance Judgment
• Evaluating Large Language Models on a Highly-specialized Topic, Radiation Oncology Physics
• "HOT" ChatGPT: The promise of ChatGPT in detecting and discriminating hateful, offensive, and toxic comments on social media
• Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Tasks
• One-Shot Labeling for Automatic Relevance Estimation
• CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation
• Testing the limits of natural language models for predicting human language judgments
• Large language models can accurately predict searcher preferences
• MIMICS-Duo: Offline & Online Evaluation of Search Clarification