WaitGPT: Monitoring and Steering Conversational LLM Agent in Data Analysis with On-the-Fly Code Visualization

Abstract

Large language models (LLMs) support data analysis through conversational user interfaces, as exemplified in OpenAI's ChatGPT (formally known as Advanced Data Analysis or Code Interpreter). Essentially, LLMs produce code for accomplishing diverse analysis tasks. However, presenting raw code can obscure the logic and hinder user verification. To empower users with enhanced comprehension and augmented control over analysis conducted by LLMs, we propose a novel approach to transform LLM-generated code into an interactive visual representation. In the approach, users are provided with a clear, step-by-step visualization of the LLM-generated code in real time, allowing them to understand, verify, and modify individual data operations in the analysis. Our design decisions are informed by a formative study (N=8) probing into user practice and challenges. We further developed a prototype named WaitGPT and conducted a user study (N=12) to evaluate its usability and effectiveness. The findings from the user study reveal that WaitGPT facilitates monitoring and steering of data analysis performed by LLMs, enabling participants to enhance error detection and increase their overall confidence in the results.

Accepted in the 37th Annual ACM Symposium on User Interface Software and Technology (UIST'24)

References in corpus (7)

• "What It Wants Me To Say": Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Models
• Spellburst: A Node-based Interface for Exploratory Creative Coding with Natural Language Prompts
• Automated Data Visualization from Natural Language via Large Language Models: An Exploratory Study
• XNLI: Explaining and Diagnosing NLI-based Visual Data Analysis
• Visualizing the Scripts of Data Wrangling with SOMNUS
• Creating Emordle: Animating Word Cloud for Emotion Expression
• Wakey-Wakey: Animate Text by Mimicking Characters in a GIF