Repurposing Annotation Guidelines to Instruct LLM Annotators: A Case Study

Abstract

This study investigates how existing annotation guidelines can be repurposed to instruct large language model (LLM) annotators for text annotation tasks. Traditional guidelines are written for human annotators who internalize training, while LLMs require explicit, structured instructions. We propose a moderation-oriented guideline repurposing method that transforms guidelines into clear directives for LLMs through an LLM moderation process. Using the NCBI Disease Corpus as a case study, our experiments show that repurposed guidelines can effectively guide LLM annotators, while revealing several practical challenges. The results highlight the potential of this workflow to support scalable and cost-effective refinement of annotation guidelines and automated annotation.

11 pages, 2 figures, 3 tables, This is a preprint of the article accepted at NLDB 2025 (Springer LNCS). The final version is available at https://doi.org/10.1007/978-3-031-97144-0_13

References in corpus (7)

• BioBERT: a pre-trained biomedical language representation model for biomedical text mining
• ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks
• BioRED: A Rich Biomedical Relation Extraction Dataset
• Can ChatGPT Reproduce Human-Generated Labels? A Study of Social Computing Tasks
• LLMs Accelerate Annotation for Medical Information Extraction
• CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation
• GoLLIE: Annotation Guidelines improve Zero-Shot Information-Extraction