Concept-Guided Chain-of-Thought Prompting for Pairwise Comparison Scoring of Texts with Large Language Models
arXiv:2310.12049 · doi:10.1109/BigData62323.2024.10825235
Abstract
Existing text scoring methods require a large corpus, struggle with short texts, or require hand-labeled data. We develop a text scoring framework that leverages generative large language models (LLMs) to (1) set texts against the backdrop of information from the near-totality of the web and digitized media, and (2) effectively transform pairwise text comparisons from a reasoning problem to a pattern recognition task. Our approach, concept-guided chain-of-thought (CGCoT), utilizes a chain of researcher-designed prompts with an LLM to generate a concept-specific breakdown for each text, akin to guidance provided to human coders. We then pairwise compare breakdowns using an LLM and aggregate answers into a score using a probability model. We apply this approach to better understand speech reflecting aversion to specific political parties on Twitter, a topic that has commanded increasing interest because of its potential contributions to democratic backsliding. We achieve stronger correlations with human judgments than widely used unsupervised text scoring methods like Wordfish. In a supervised setting, besides a small pilot dataset to develop CGCoT prompts, our measures require no additional hand-labeled data and produce predictions on par with RoBERTa-Large fine-tuned on thousands of hand-labeled tweets. This project showcases the potential of combining human expertise and LLMs for scoring tasks.
10 pages, 2 figures. Appears in 2024 IEEE International Conference on Big Data (BigData). Please cite the published version: 10.1109/BigData62323.2024.10825235
References in corpus (10)
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
- Large Language Models are Zero-Shot Reasoners
- ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks
- The Geometry of Culture: Analyzing Meaning through Word Embeddings
- Least-to-Most Prompting Enables Complex Reasoning in Large Language Models
- ChatGPT-4 Outperforms Experts and Crowd Workers in Annotating Political Twitter Messages with Zero-Shot Learning
- FrameAxis: Characterizing Microframe Bias and Intensity with Word Embedding
- Large Language Models Can Be Used to Estimate the Latent Positions of Politicians
- AI Chat Assistants can Improve Conversations about Divisive Topics
- Measurement in the Age of LLMs: An Application to Ideological Scaling