ChartGPT: Leveraging LLMs to Generate Charts from Abstract Natural Language

Abstract

The use of natural language interfaces (NLIs) to create charts is becoming increasingly popular due to the intuitiveness of natural language interactions. One key challenge in this approach is to accurately capture user intents and transform them to proper chart specifications. This obstructs the wide use of NLI in chart generation, as users' natural language inputs are generally abstract (i.e., ambiguous or under-specified), without a clear specification of visual encodings. Recently, pre-trained large language models (LLMs) have exhibited superior performance in understanding and generating natural language, demonstrating great potential for downstream tasks. Inspired by this major trend, we propose ChartGPT, generating charts from abstract natural language inputs. However, LLMs are struggling to address complex logic problems. To enable the model to accurately specify the complex parameters and perform operations in chart generation, we decompose the generation process into a step-by-step reasoning pipeline, so that the model only needs to reason a single and specific sub-task during each run. Moreover, LLMs are pre-trained on general datasets, which might be biased for the task of chart generation. To provide adequate visualization knowledge, we create a dataset consisting of abstract utterances and charts and improve model performance through fine-tuning. We further design an interactive interface for ChartGPT that allows users to check and modify the intermediate outputs of each step. The effectiveness of the proposed system is evaluated through quantitative evaluations and a user study.

References in corpus (14)

• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• LLaMA: Open and Efficient Foundation Language Models
• Scaling Instruction-Finetuned Language Models
• Emergent Abilities of Large Language Models
• Least-to-Most Prompting Enables Complex Reasoning in Large Language Models
• Neural Text Generation with Unlikelihood Training
• Towards Natural Language Interfaces for Data Visualization: A Survey
• Augmented Language Models: a Survey
• KG4Vis: A Knowledge Graph-Based Approach for Visualization Recommendation
• Collecting and Characterizing Natural Language Utterances for Specifying Data Visualizations
• Visualizing the Scripts of Data Wrangling with SOMNUS
• Reviving Static Charts into Live Charts
• Revealing the Semantics of Data Wrangling Scripts With COMANTICS
• Data-Copilot: Bridging Billions of Data and Humans with Autonomous Workflow

Cited by in corpus (11)

• WaitGPT: Monitoring and Steering Conversational LLM Agent in Data Analysis with On-the-Fly Code Visualization
• LightVA: Lightweight Visual Analytics with LLM Agent-Based Task Planning and Execution
• Reviving Static Charts into Live Charts
• InstructPipe: Generating Visual Blocks Pipelines with Human Instructions and LLMs
• LLM-Assisted Visual Analytics: Opportunities and Challenges
• Evaluating LLMs for Visualization Generation and Understanding
• InterChat: Enhancing Generative Visual Analytics using Multimodal Interactions
• DeepVIS: Bridging Natural Language and Data Visualization Through Step-wise Reasoning
• ReSpark: Leveraging Previous Data Reports as References to Generate New Reports with LLMs
• Making Multimodal LLMs Reliable Chart Data Extractors: A Benchmark and Training Framework
• Evaluating Prompting Strategies for Chart Question Answering with Large Language Models