InstructPipe: Generating Visual Blocks Pipelines with Human Instructions and LLMs
arXiv:2312.09672 · doi:10.1145/3706598.3713905
Abstract
Visual programming has the potential of providing novice programmers with a low-code experience to build customized processing pipelines. Existing systems typically require users to build pipelines from scratch, implying that novice users are expected to set up and link appropriate nodes from a blank workspace. In this paper, we introduce InstructPipe, an AI assistant for prototyping machine learning (ML) pipelines with text instructions. We contribute two large language model (LLM) modules and a code interpreter as part of our framework. The LLM modules generate pseudocode for a target pipeline, and the interpreter renders the pipeline in the node-graph editor for further human-AI collaboration. Both technical and user evaluation (N=16) shows that InstructPipe empowers users to streamline their ML pipeline workflow, reduce their learning curve, and leverage open-ended commands to spark innovative ideas.
CHI 2025
References in corpus (17)
- Array Programming with NumPy
- NL4DV: A Toolkit for Generating Analytic Specifications for Data Visualization from Natural Language Queries
- Towards Natural Language Interfaces for Data Visualization: A Survey
- ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing
- Graphologue: Exploring Large Language Model Responses with Interactive Diagrams
- ChartGPT: Leveraging LLMs to Generate Charts from Abstract Natural Language
- VISAR: A Human-AI Argumentative Writing Assistant with Visual Programming and Rapid Draft Prototyping
- Towards Natural Language-Based Visualization Authoring
- How to Teach Programming in the AI Era? Using LLMs as a Teachable Agent for Debugging
- How Beginning Programmers and Code LLMs (Mis)read Each Other
- ChatScratch: An AI-Augmented System Toward Autonomous Visual Programming Learning for Children Aged 6-12
- VRCopilot: Authoring 3D Layouts with Generative AI Models in VR
- Does BLEU Score Work for Code Migration?
- Storyfier: Exploring Vocabulary Learning Support with Text Generation Models
- Gesture-aware Interactive Machine Teaching with In-situ Object Annotations
- Visual Programming: Compositional visual reasoning without training
- Facilitating Conversational Interaction in Natural Language Interfaces for Visualization