Towards Natural Language-Based Visualization Authoring
arXiv:2208.10947 · doi:10.1109/TVCG.2022.3209357
Abstract
A key challenge to visualization authoring is the process of getting familiar with the complex user interfaces of authoring tools. Natural Language Interface (NLI) presents promising benefits due to its learnability and usability. However, supporting NLIs for authoring tools requires expertise in natural language processing, while existing NLIs are mostly designed for visual analytic workflow. In this paper, we propose an authoring-oriented NLI pipeline by introducing a structured representation of users' visualization editing intents, called editing actions, based on a formative study and an extensive survey on visualization construction tools. The editing actions are executable, and thus decouple natural language interpretation and visualization applications as an intermediate layer. We implement a deep learning-based NL interpreter to translate NL utterances into editing actions. The interpreter is reusable and extensible across authoring tools. The authoring tools only need to map the editing actions into tool-specific operations. To illustrate the usages of the NL interpreter, we implement an Excel chart editor and a proof-of-concept authoring tool, VisTalk. We conduct a user study with VisTalk to understand the usage patterns of NL-based authoring systems. Finally, we discuss observations on how users author charts with natural language, as well as implications for future research.
References in corpus (11)
- Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
- NL4DV: A Toolkit for Generating Analytic Specifications for Data Visualization from Natural Language Queries
- Towards Natural Language Interfaces for Data Visualization: A Survey
- A Comparative Survey of Recent Natural Language Interfaces for Databases
- FlowSense: A Natural Language Interface for Visual Data Exploration within a Dataflow System
- Critical Reflections on Visualization Authoring Systems
- Touch? Speech? or Touch and Speech? Investigating Multimodal Interaction for Visual Network Exploration and Analysis
- Collecting and Characterizing Natural Language Utterances for Specifying Data Visualizations
- Interweaving Multimodal Interaction with Flexible Unit Visualizations for Data Exploration
- Studying Visualization Guidelines According to Grounded Theory
- VisQualdex -- the comprehensive guide to good data visualization
Cited by in corpus (14)
- TypeDance: Creating Semantic Typographic Logos from Image through Personalized Generation
- XNLI: Explaining and Diagnosing NLI-based Visual Data Analysis
- Natural Language Dataset Generation Framework for Visualizations Powered by Large Language Models
- Narrative Player: Reviving Data Narratives with Visuals
- NotePlayer: Engaging Jupyter Notebooks for Dynamic Presentation of Analytical Processes
- PyGWalker: On-the-fly Assistant for Exploratory Visual Data Analysis
- InstructPipe: Generating Visual Blocks Pipelines with Human Instructions and LLMs
- LLMs as Workers in Human-Computational Algorithms? Replicating Crowdsourcing Pipelines with LLMs
- InterChat: Enhancing Generative Visual Analytics using Multimodal Interactions
- VOICE: Visual Oracle for Interaction, Conversation, and Explanation
- DeepVIS: Bridging Natural Language and Data Visualization Through Step-wise Reasoning
- PaperBridge: Crafting Research Narratives through Human-AI Co-Exploration
- Natural Language Interaction for Editing Visual Knowledge Graphs
- Chart What I Say: Exploring Cross-Modality Prompt Alignment in AI-Assisted Chart Authoring