Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding

Abstract

Qualitative analysis of textual contents unpacks rich and valuable information by assigning labels to the data. However, this process is often labor-intensive, particularly when working with large datasets. While recent AI-based tools demonstrate utility, researchers may not have readily available AI resources and expertise, let alone be challenged by the limited generalizability of those task-specific models. In this study, we explored the use of large language models (LLMs) in supporting deductive coding, a major category of qualitative analysis where researchers use pre-determined codebooks to label the data into a fixed set of codes. Instead of training task-specific models, a pre-trained LLM could be used directly for various tasks without fine-tuning through prompt learning. Using a curiosity-driven questions coding task as a case study, we found, by combining GPT-3 with expert-drafted codebooks, our proposed approach achieved fair to substantial agreements with expert-coded results. We lay out challenges and opportunities in using LLMs to support qualitative coding and beyond.

arXiv comment28th International Conference on Intelligent User Interfaces (IUI '23 Companion), March 27--31, 2023, Sydney, NSW, Australia

References in corpus (3)

• PaLM: Scaling Language Modeling with Pathways
• Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing
• Conversational agents for fostering curiosity-driven learning in children

Cited by in corpus (16)

• Harnessing ChatGPT for thematic analysis: Are we ready?
• Generative Artificial Intelligence in Learning Analytics: Contextualising Opportunities and Challenges through the Learning Analytics Cycle
• Redefining Qualitative Analysis in the AI Era: Utilizing ChatGPT for Efficient Thematic Analysis
• A Taxonomy for Human-LLM Interaction Modes: An Initial Exploration
• Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LLooM
• Predicting challenge moments from students' discourse: A comparison of GPT-4 to two traditional natural language processing approaches
• Embedding Democratic Values into Social Media AIs via Societal Objective Functions
• ContextQ: Generated Questions to Support Meaningful Parent-Child Dialogue While Co-Reading
• Silent Guardian: Protecting Text from Malicious Exploitation by Large Language Models
• Harnessing LLMs for Automated Video Content Analysis: An Exploratory Workflow of Short Videos on Depression
• From Voices to Validity: Leveraging Large Language Models (LLMs) for Textual Analysis of Policy Stakeholder Interviews
• Can Large Language Models Grasp Concepts in Visual Content? A Case Study on YouTube Shorts about Depression
• The Homework Wars: Exploring Emotions, Behaviours, and Conflicts in Parent-Child Homework Interactions
• "Mango Mango, How to Let The Lettuce Dry Without A Spinner?": Exploring User Perceptions of Using An LLM-Based Conversational Assistant Toward Cooking Partner
• Bridging Dictionary: AI-Generated Dictionary of Partisan Language Use
• Policy Maps: Tools for Guiding the Unbounded Space of LLM Behaviors