DirectGPT: A Direct Manipulation Interface to Interact with Large Language Models

Abstract

We characterize and demonstrate how the principles of direct manipulation can improve interaction with large language models. This includes: continuous representation of generated objects of interest; reuse of prompt syntax in a toolbar of commands; manipulable outputs to compose or control the effect of prompts; and undo mechanisms. This idea is exemplified in DirectGPT, a user interface layer on top of ChatGPT that works by transforming direct manipulation actions to engineered prompts. A study shows participants were 50% faster and relied on 50% fewer and 72% shorter prompts to edit text, code, and vector images compared to baseline ChatGPT. Our work contributes a validated approach to integrate LLMs into traditional software using direct manipulation. Data, code, and demo available at https://osf.io/3wt6s.

arXiv commentIn Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (CHI '24)

References in corpus (14)

• SciPy 1.0--Fundamental Algorithms for Scientific Computing in Python
• Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold
• Sensecape: Enabling Multilevel Exploration and Sensemaking with Large Language Models
• ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing
• Graphologue: Exploring Large Language Model Responses with Interactive Diagrams
• "What It Wants Me To Say": Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Models
• Friend, Collaborator, Student, Manager: How Design of an AI-Driven Game Level Editor Affects Creators
• PromptPaint: Steering Text-to-Image Generation Through Paint Medium-like Interactions
• Choice Over Control: How Users Write with Large Language Models using Diegetic and Non-Diegetic Prompting
• Spellburst: A Node-based Interface for Exploratory Creative Coding with Natural Language Prompts
• GANSlider: How Users Control Generative Models for Images using Multiple Sliders with and without Feedforward Information
• Fast Inference from Transformers via Speculative Decoding
• Suggestion Lists vs. Continuous Generation: Interaction Design for Writing with Generative Models on Mobile Devices Affect Text Length, Wording and Perceived Authorship
• Improving GAN Equilibrium by Raising Spatial Awareness

Cited by in corpus (24)

• WaitGPT: Monitoring and Steering Conversational LLM Agent in Data Analysis with On-the-Fly Code Visualization
• LightVA: Lightweight Visual Analytics with LLM Agent-Based Task Planning and Execution
• AI-Instruments: Embodying Prompts as Instruments to Abstract & Reflect Graphical Interface Commands as General-Purpose Tools
• Misty: UI Prototyping Through Interactive Conceptual Blending
• The Value-Sensitive Conversational Agent Co-Design Framework
• Writing with AI Lowers Psychological Ownership, but Longer Prompts Can Help
• Textoshop: Interactions Inspired by Drawing Software to Facilitate Text Editing
• Intent Tagging: Exploring Micro-Prompting Interactions for Supporting Granular Human-GenAI Co-Creation Workflows
• SPROUT: an Interactive Authoring Tool for Generating Programming Tutorials with the Visualization of Large Language Models
• NeuroSync: Intent-Aware Code-Based Problem Solving via Direct LLM Understanding Modification
• OnGoal: Tracking and Visualizing Conversational Goals in Multi-Turn Dialogue with Large Language Models
• Composable Prompting Workspaces for Creative Writing: Exploration and Iteration Using Dynamic Widgets
• GenPara: Enhancing the 3D Design Editing Process by Inferring Users' Regions of Interest with Text-Conditional Shape Parameters
• TrICy: Trigger-guided Data-to-text Generation with Intent aware Attention-Copy
• InterChat: Enhancing Generative Visual Analytics using Multimodal Interactions
• StoryEnsemble: Enabling Dynamic Exploration & Iteration in the Design Process with AI and Forward-Backward Propagation
• Exploring Mobile Touch Interaction with Large Language Models
• PosterMate: Audience-driven Collaborative Persona Agents for Poster Design
• KinemaFX: A Kinematic-Driven Interactive System for Particle Effect Exploration and Customization
• Visual Story-Writing: Writing by Manipulating Visual Representations of Stories
• How to Steer Your Multi-Agent System: Human-LLM Collaborative Planning
• From Planning to Revision: How AI Writing Support at Different Stages Alters Ownership
• StudyAlign: A Software System for Conducting Web-Based User Studies with Functional Interactive Prototypes
• OOPrompt: Reifying Intents into Structured Artifacts for Modular and Iterative Prompting