WaitGPT: Monitoring and Steering Conversational LLM Agent in Data Analysis with On-the-Fly Code Visualization

Abstract

Large language models (LLMs) support data analysis through conversational user interfaces, as exemplified in OpenAI's ChatGPT (formally known as Advanced Data Analysis or Code Interpreter). Essentially, LLMs produce code for accomplishing diverse analysis tasks. However, presenting raw code can obscure the logic and hinder user verification. To empower users with enhanced comprehension and augmented control over analysis conducted by LLMs, we propose a novel approach to transform LLM-generated code into an interactive visual representation. In the approach, users are provided with a clear, step-by-step visualization of the LLM-generated code in real time, allowing them to understand, verify, and modify individual data operations in the analysis. Our design decisions are informed by a formative study (N=8) probing into user practice and challenges. We further developed a prototype named WaitGPT and conducted a user study (N=12) to evaluate its usability and effectiveness. The findings from the user study reveal that WaitGPT facilitates monitoring and steering of data analysis performed by LLMs, enabling participants to enhance error detection and increase their overall confidence in the results.

Accepted in the 37th Annual ACM Symposium on User Interface Software and Technology (UIST'24)

References in corpus (14)

• The Metacognitive Demands and Opportunities of Generative AI
• Towards Natural Language Interfaces for Data Visualization: A Survey
• Sensecape: Enabling Multilevel Exploration and Sensemaking with Large Language Models
• Graphologue: Exploring Large Language Model Responses with Interactive Diagrams
• "What It Wants Me To Say": Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Models
• ChartGPT: Leveraging LLMs to Generate Charts from Abstract Natural Language
• DirectGPT: A Direct Manipulation Interface to Interact with Large Language Models
• Spellburst: A Node-based Interface for Exploratory Creative Coding with Natural Language Prompts
• CrossData: Leveraging Text-Data Connections for Authoring Data Documents
• Automated Data Visualization from Natural Language via Large Language Models: An Exploratory Study
• XNLI: Explaining and Diagnosing NLI-based Visual Data Analysis
• Visualizing the Scripts of Data Wrangling with SOMNUS
• Creating Emordle: Animating Word Cloud for Emotion Expression
• Wakey-Wakey: Animate Text by Mimicking Characters in a GIF

Cited by in corpus (5)

• AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges
• A Survey on Large Language Model-based Agents for Statistics and Data Science
• NeuroSync: Intent-Aware Code-Based Problem Solving via Direct LLM Understanding Modification
• RAGTrace: Understanding and Refining Retrieval-Generation Dynamics in Retrieval-Augmented Generation
• DeepVIS: Bridging Natural Language and Data Visualization Through Step-wise Reasoning