"What It Wants Me To Say": Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Models

Abstract

Code-generating large language models translate natural language into code. However, only a small portion of the infinite space of naturalistic utterances is effective at guiding code generation. For non-expert end-user programmers, learning this is the challenge of abstraction matching. We examine this challenge in the specific context of data analysis in spreadsheets, in a system that maps the users natural language query to Python code using the Codex generator, executes the code, and shows the result. We propose grounded abstraction matching, which bridges the abstraction gap by translating the code back into a systematic and predictable naturalistic utterance. In a between-subjects, think-aloud study (n=24), we compare grounded abstraction matching to an ungrounded alternative based on previously established query framing principles. We find that the grounded approach improves end-users' understanding of the scope and capabilities of the code-generating model, and the kind of language needed to use it effectively.

References in corpus (9)

• Evaluating Large Language Models Trained on Code
• Competition-Level Code Generation with AlphaCode
• LaMDA: Language Models for Dialog Applications
• Perfection Not Required? Human-AI Partnerships in Code Translation
• What is it like to program with artificial intelligence?
• TableSense: Spreadsheet Table Detection with Convolutional Neural Networks
• Crystalline: Lowering the Cost for Developers to Collect and Organize Information for Decision Making
• Understanding How Programmers Can Use Annotations on Documentation
• Automatic Code Documentation Generation Using GPT-3

Cited by in corpus (27)

• The Metacognitive Demands and Opportunities of Generative AI
• DirectGPT: A Direct Manipulation Interface to Interact with Large Language Models
• EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteria
• "We Need Structured Output": Towards User-centered Constraints on Large Language Model Output
• How Beginning Programmers and Code LLMs (Mis)read Each Other
• A Taxonomy for Human-LLM Interaction Modes: An Initial Exploration
• Improving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decomposition
• WaitGPT: Monitoring and Steering Conversational LLM Agent in Data Analysis with On-the-Fly Code Visualization
• Selenite: Scaffolding Online Sensemaking with Comprehensive Overviews Elicited from Large Language Models
• Deploying and Evaluating LLMs to Program Service Mobile Robots
• LessonPlanner: Assisting Novice Teachers to Prepare Pedagogy-Driven Lesson Plans with Large Language Models
• What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Use
• Exploring How Multiple Levels of GPT-Generated Programming Hints Support or Disappoint Novices
• Beyond Code Generation: LLM-supported Exploration of the Program Design Space
• "It's like a rubber duck that talks back": Understanding Generative AI-Assisted Data Analysis Workflows through a Participatory Prompting Study
• CoPrompt: Supporting Prompt Sharing and Referring in Collaborative Natural Language Programming
• Will Code Remain a Relevant User Interface for End-User Programming with Generative AI Models?
• To Search or To Gen? Exploring the Synergy between Generative AI and Web Search in Programming
• DesignWeaver: Dimensional Scaffolding for Text-to-Image Product Design
• Classroom Simulacra: Building Contextual Student Generative Agents in Online Education for Learning Behavioral Simulation
• JailbreakLens: Visual Analysis of Jailbreak Attacks Against Large Language Models
• NeuroSync: Intent-Aware Code-Based Problem Solving via Direct LLM Understanding Modification
• Intent Tagging: Exploring Micro-Prompting Interactions for Supporting Granular Human-GenAI Co-Creation Workflows
• Gensors: Authoring Personalized Visual Sensors with Multimodal Foundation Models and Reasoning
• LLMs as Workers in Human-Computational Algorithms? Replicating Crowdsourcing Pipelines with LLMs
• GenComUI: Exploring Generative Visual Aids as Medium to Support Task-Oriented Human-Robot Communication
• Inline Visualization and Manipulation of Real-Time Hardware Log for Supporting Debugging of Embedded Programs