Prompt Design Matters for Computational Social Science Tasks but in Unpredictable Ways

Abstract

Manually annotating data for computational social science tasks can be costly, time-consuming, and emotionally draining. While recent work suggests that LLMs can perform such annotation tasks in zero-shot settings, little is known about how prompt design impacts LLMs' compliance and accuracy. We conduct a large-scale multi-prompt experiment to test how model selection (ChatGPT, PaLM2, and Falcon7b) and prompt design features (definition inclusion, output type, explanation, and prompt length) impact the compliance and accuracy of LLM-generated annotations on four CSS tasks (toxicity, sentiment, rumor stance, and news frames). Our results show that LLM compliance and accuracy are highly prompt-dependent. For instance, prompting for numerical scores instead of labels reduces all LLMs' compliance and accuracy. The overall best prompting setup is task-dependent, and minor prompt changes can cause large changes in the distribution of generated labels. By showing that prompt design significantly impacts the quality and distribution of LLM-generated annotations, this work serves as both a warning and practical guide for researchers and practitioners.

under review

References in corpus (9)

• Tree of Thoughts: Deliberate Problem Solving with Large Language Models
• CrowdWorkSheets: Accounting for Individual and Collective Identities Underlying Crowdsourced Dataset Annotation
• Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them
• ChatGPT Prompt Patterns for Improving Code Quality, Refactoring, Requirements Elicitation, and Software Design
• GraphPrompt: Unifying Pre-Training and Downstream Tasks for Graph Neural Networks
• Applying Large Language Models and Chain-of-Thought for Automatic Scoring
• Understanding Journalists' Workflows in News Curation
• Watch Your Language: Investigating Content Moderation with Large Language Models
• Stance Detection with Collaborative Role-Infused LLM-Based Agents

Cited by in corpus (2)

• Codebook LLMs: Evaluating LLMs as Measurement Tools for Political Science Concepts
• Evaluating Large Language Models for Antisemitic Incident Classification