Towards Natural Language Interfaces for Data Visualization: A Survey

Abstract

Utilizing Visualization-oriented Natural Language Interfaces (V-NLI) as a complementary input modality to direct manipulation for visual analytics can provide an engaging user experience. It enables users to focus on their tasks rather than having to worry about how to operate visualization tools on the interface. In the past two decades, leveraging advanced natural language processing technologies, numerous V-NLI systems have been developed in academic research and commercial software, especially in recent years. In this article, we conduct a comprehensive review of the existing V-NLIs. In order to classify each paper, we develop categorical dimensions based on a classic information visualization pipeline with the extension of a V-NLI layer. The following seven stages are used: query interpretation, data transformation, visual mapping, view transformation, human interaction, dialogue management, and presentation. Finally, we also shed light on several promising directions for future work in the V-NLI community.

20 pages, 15 figures, accepted by IEEE TVCG

References in corpus (31)

• Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
• TAPAS: Weakly Supervised Table Parsing via Pre-training
• SQLNet: Generating Structured Queries From Natural Language Without Reinforcement Learning
• NL4DV: A Toolkit for Generating Analytic Specifications for Data Visualization from Natural Language Queries
• Evaluation of Text Generation: A Survey
• Calliope: Automatic Visual Data Story Generation from a Spreadsheet
• A Survey on ML4VIS: Applying Machine Learning Advances to Data Visualization
• KG4Vis: A Knowledge Graph-Based Approach for Visualization Recommendation
• Data Visceralization: Enabling Deeper Understanding of Data Using Virtual Reality
• Annotating Columns with Pre-trained Language Models
• Augmenting Static Visualizations with PapARVis Designer
• Touch? Speech? or Touch and Speech? Investigating Multimodal Interaction for Visual Network Exploration and Analysis
• Data@Hand: Fostering Visual Exploration of Personal Data on Smartphones Leveraging Speech and Touch Interaction
• Bridging the Semantic Gap with SQL Query Logs in Natural Language Interfaces to Databases
• Collecting and Characterizing Natural Language Utterances for Specifying Data Visualizations
• TURL: Table Understanding through Representation Learning
• The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
• Towards Understanding How Readers Integrate Charts and Captions: A Case Study with Line Charts
• Coupling Story to Visualization: Using Textual Analysis as a Bridge Between Data and Interpretation
• Interweaving Multimodal Interaction with Flexible Unit Visualizations for Data Exploration
• Table2Charts: Recommending Charts by Learning Shared Table Representations
• Falx: Synthesis-Powered Visualization Authoring
• Understanding Infographics through Textual and Visual Tag Prediction
• "Hey Model!" - Natural User Interactions and Agency in Accessible Interactive 3D Models
• Task-Based Effectiveness of Interactive Contiguous Area Cartograms
• Foresight: Recommending Visual Insights
• Commonsense Knowledge Enhanced Embeddings for Solving Pronoun Disambiguation Problems in Winograd Schema Challenge
• Quda: Natural Language Queries for Visual Data Analytics
• Insight-centric Visualization Recommendation
• A Brief Survey and Comparative Study of Recent Development of Pronoun Coreference Resolution
• DCoM: A Deep Column Mapper for Semantic Data Type Detection

Cited by in corpus (26)

• ChartGPT: Leveraging LLMs to Generate Charts from Abstract Natural Language
• Towards Natural Language-Based Visualization Authoring
• LEVA: Using Large Language Models to Enhance Visual Analytics
• A Survey of Text-to-SQL in the Era of LLMs: Where are we, and where are we going?
• WaitGPT: Monitoring and Steering Conversational LLM Agent in Data Analysis with On-the-Fly Code Visualization
• XNLI: Explaining and Diagnosing NLI-based Visual Data Analysis
• Pair-Level Supervised Contrastive Learning for Natural Language Inference
• HAIChart: Human and AI Paired Visualization System
• Explainable XR: Understanding User Behaviors of XR Environments using LLM-assisted Analytics Framework
• Visual Data Analysis with Task-based Recommendations
• Natural Language Dataset Generation Framework for Visualizations Powered by Large Language Models
• RASIPAM: Interactive Pattern Mining of Multivariate Event Sequences in Racket Sports
• This is the Table I Want! Interactive Data Transformation on Desktop and in Virtual Reality
• Narrative Player: Reviving Data Narratives with Visuals
• NotePlayer: Engaging Jupyter Notebooks for Dynamic Presentation of Analytical Processes
• Too Many Cooks: Exploring How Graphical Perception Studies Influence Visualization Recommendations in Draco
• PyGWalker: On-the-fly Assistant for Exploratory Visual Data Analysis
• NeuroSync: Intent-Aware Code-Based Problem Solving via Direct LLM Understanding Modification
• InstructPipe: Generating Visual Blocks Pipelines with Human Instructions and LLMs
• LLM-Assisted Visual Analytics: Opportunities and Challenges
• Evaluating LLMs for Visualization Generation and Understanding
• InterChat: Enhancing Generative Visual Analytics using Multimodal Interactions
• Vital Insight: Assisting Experts' Context-Driven Sensemaking of Multi-modal Personal Tracking Data Using Visualization and Human-In-The-Loop LLM
• Towards Automated Cross-domain Exploratory Data Analysis through Large Language Models
• DeepVIS: Bridging Natural Language and Data Visualization Through Step-wise Reasoning
• Natural Language Interaction for Editing Visual Knowledge Graphs