Improving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decomposition

Abstract

LLM-powered tools like ChatGPT Data Analysis, have the potential to help users tackle the challenging task of data analysis programming, which requires expertise in data processing, programming, and statistics. However, our formative study (n=15) uncovered serious challenges in verifying AI-generated results and steering the AI (i.e., guiding the AI system to produce the desired output). We developed two contrasting approaches to address these challenges. The first (Stepwise) decomposes the problem into step-by-step subgoals with pairs of editable assumptions and code until task completion, while the second (Phasewise) decomposes the entire problem into three editable, logical phases: structured input/output assumptions, execution plan, and code. A controlled, within-subjects experiment (n=18) compared these systems against a conversational baseline. Users reported significantly greater control with the Stepwise and Phasewise systems, and found intervention, correction, and verification easier, compared to the baseline. The results suggest design guidelines and trade-offs for AI-assisted data analysis tools.

Published at UIST 2024; 19 pages, 9 figures, and 2 tables

References in corpus (14)

• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• Questioning the AI: Informing Design Practices for Explainable AI User Experiences
• The Metacognitive Demands and Opportunities of Generative AI
• CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needs
• "Help Me Help the AI": Understanding How Explainability Can Support Human-AI Interaction
• Conceptual Metaphors Impact Perceptions of Human-AI Collaboration
• Sensecape: Enabling Multilevel Exploration and Sensemaking with Large Language Models
• "What It Wants Me To Say": Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Models
• AutoDS: Towards Human-Centered Automation of Data Science
• Talking datasets: Understanding data sensemaking behaviours
• Paths Explored, Paths Omitted, Paths Obscured: Decision Points & Selective Reporting in End-to-End Data Analysis
• "It's like a rubber duck that talks back": Understanding Generative AI-Assisted Data Analysis Workflows through a Participatory Prompting Study
• Visualizing the Scripts of Data Wrangling with SOMNUS
• Should Computers Be Easy To Use? Questioning the Doctrine of Simplicity in User Interface Design

Cited by in corpus (4)

• Beyond Code Generation: LLM-supported Exploration of the Program Design Space
• ChainBuddy: An AI Agent System for Generating LLM Pipelines
• Human-AI Experience in Integrated Development Environments: A Systematic Literature Review
• Do It For Me vs. Do It With Me: Investigating User Perceptions of Different Paradigms of Automation in Copilots for Feature-Rich Software