Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LLooM

Abstract

Data analysts have long sought to turn unstructured text data into meaningful concepts. Though common, topic modeling and clustering focus on lower-level keywords and require significant interpretative work. We introduce concept induction, a computational process that instead produces high-level concepts, defined by explicit inclusion criteria, from unstructured text. For a dataset of toxic online comments, where a state-of-the-art BERTopic model outputs "women, power, female," concept induction produces high-level concepts such as "Criticism of traditional gender roles" and "Dismissal of women's concerns." We present LLooM, a concept induction algorithm that leverages large language models to iteratively synthesize sampled text and propose human-interpretable concepts of increasing generality. We then instantiate LLooM in a mixed-initiative text analysis tool, enabling analysts to shift their attention from interpreting topics to engaging in theory-driven analysis. Through technical evaluations and four analysis scenarios ranging from literature review to content moderation, we find that LLooM's concepts improve upon the prior art of topic models in terms of quality and data coverage. In expert case studies, LLooM helped researchers to uncover new insights even from familiar datasets, for example by suggesting a previously unnoticed concept of attacks on out-party stances in a political social media dataset.

To appear at CHI 2024

References in corpus (12)

• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding
• Sensecape: Enabling Multilevel Exploration and Sensemaking with Large Language Models
• Breaking Out of the Ivory Tower: A Large-scale Analysis of Patent Citations to HCI Research
• Unpacking the Expressed Consequences of AI Research in Broader Impact Statements
• Embedding Democratic Values into Social Media AIs via Societal Objective Functions
• Model Sketching: Centering Concepts in Early-Stage Machine Learning Model Design
• Scholastic: Graphical Human-Al Collaboration for Inductive and Interpretive Text Analysis
• Tisane: Authoring Statistical Models via Formal Reasoning from Conceptual and Data Relationships
• Is Automated Topic Model Evaluation Broken?: The Incoherence of Coherence
• GANzilla: User-Driven Direction Discovery in Generative Adversarial Networks
• Semantic Concept Spaces: Guided Topic Model Refinement using Word-Embedding Projections

Cited by in corpus (11)

• Beyond Code Generation: LLM-supported Exploration of the Program Design Space
• Thoughtful Adoption of NLP for Civic Participation: Understanding Differences Among Policymakers
• LLLMs: A Data-Driven Survey of Evolving Research on Limitations of Large Language Models
• ScholarMate: A Mixed-Initiative Tool for Qualitative Knowledge Work and Information Sensemaking
• Toward Human-Quantum Computer Interaction: Interface Techniques for Usable Quantum Computing
• Divisi: Interactive Search and Visualization for Scalable Exploratory Subgroup Analysis
• Clarify: Improving Model Robustness With Natural Language Corrections
• PolicyPulse: LLM-Synthesis Tool for Policy Researchers
• DiscipLink: Unfolding Interdisciplinary Information Seeking Process via Human-AI Co-Exploration
• Policy Maps: Tools for Guiding the Unbounded Space of LLM Behaviors
• Beyond the Lens: Quantifying the Impact of Scientific Documentaries through Amazon Reviews