Improving Text-to-SQL Evaluation Methodology

Abstract

To be informative, an evaluation must measure how well systems generalize to realistic unseen data. We identify limitations of and propose improvements to current evaluations of text-to-SQL systems. First, we compare human-generated and automatically generated questions, characterizing properties of queries necessary for real-world applications. To facilitate evaluation on multiple datasets, we release standardized and improved versions of seven existing datasets and one new text-to-SQL dataset. Second, we show that the current division of data into training and test sets measures robustness to variations in the way questions are asked, but only partially tests how well systems generalize to new queries; therefore, we propose a complementary dataset split for evaluation of future work. Finally, we demonstrate how the common practice of anonymizing variables during evaluation removes an important challenge of the task. Our observations highlight key difficulties, and our methodology enables effective measurement of future development.

To appear at ACL 2018

References in corpus (10)

• Neural Machine Translation by Jointly Learning to Align and Translate
• Sequence to Sequence Learning with Neural Networks
• Learning to Map Sentences to Logical Form: Structured Classification with Probabilistic Categorial Grammars
• Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
• DyNet: The Dynamic Neural Network Toolkit
• Learning Dependency-Based Compositional Semantics
• Massive Exploration of Neural Machine Translation Architectures
• Learning a Natural Language Interface with Neural Programmer
• Type- and Content-Driven Synthesis of SQL Queries from Natural Language
• An Encoder-Decoder Framework Translating Natural Language to Database Queries

Cited by in corpus (60)

• Database Meets Deep Learning: Challenges and Opportunities
• Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architectures
• Measuring Compositional Generalization: A Comprehensive Method on Realistic Data
• X-SQL: reinforce schema representation with context
• Unlocking Compositional Generalization in Pre-trained Models Using Intermediate Representations
• A Survey of Text-to-SQL in the Era of LLMs: Where are we, and where are we going?
• Editing-Based SQL Query Generation for Cross-Domain Context-Dependent Questions
• Grammar-based Neural Text-to-SQL Generation
• CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases
• Towards Ecologically Valid Research on Language User Interfaces
• Representing Schema Structure with Graph Neural Networks for Text-to-SQL Parsing
• Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representation
• SpatialNLI: A Spatial Domain Natural Language Interface to Databases Using Spatial Comprehension
• Scalable Neural Methods for Reasoning With a Symbolic Knowledge Base
• Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task
• TableQA: a Large-Scale Chinese Text-to-SQL Dataset for Table-Aware SQL Generation
• Knowledge Graph-based Question Answering with Electronic Health Records
• Structured Reordering for Modeling Latent Alignments in Sequence Transduction
• Quda: Natural Language Queries for Visual Data Analytics
• One-Shot Learning for Text-to-SQL Generation
• Semantic Evaluation for Text-to-SQL with Distilled Test Suites
• Neural Semantic Parsing with Anonymization for Command Understanding in General-Purpose Service Robots
• Photon: A Robust Cross-Domain Text-to-SQL System
• Text-to-SQL Generation for Question Answering on Electronic Medical Records
• Did You Ask a Good Question? A Cross-Domain Question Intention Classification Benchmark for Text-to-SQL
• Text-to-SQL in the Wild: A Naturally-Occurring Dataset Based on Stack Exchange Data
• ColloQL: Robust Cross-Domain Text-to-SQL Over Search Queries
• Compositional Generalization via Semantic Tagging
• Text-to-SQL Domain Adaptation via Human-LLM Collaborative Data Annotation
• Latent Compositional Representations Improve Systematic Generalization in Grounded Question Answering
• FeTaQA: Free-form Table Question Answering
• On the Potential of Lexico-logical Alignments for Semantic Parsing to SQL Queries
• Learning Semantic Parsers from Denotations with Latent Structured Alignments and Abstract Programs
• Zero-shot Text-to-SQL Learning with Auxiliary Task
• Meta-Learning for Domain Generalization in Semantic Parsing
• Sequence-Level Mixed Sample Data Augmentation
• An Investigation Between Schema Linking and Text-to-SQL Performance
• Learning to Synthesize Data for Semantic Parsing
• Code Generation from Natural Language with Less Prior and More Monolingual Data
• ScienceBenchmark: A Complex Real-World Benchmark for Evaluating Natural Language to SQL Systems
• Prefix-to-SQL: Text-to-SQL Generation from Incomplete User Questions
• Tracking Interaction States for Multi-Turn Text-to-SQL Semantic Parsing
• A Pilot Study for Chinese SQL Semantic Parsing
• Few-Shot Semantic Parsing for New Predicates
• Byte-Pair Encoding for Text-to-SQL Generation
• Semantic Parsing to Manipulate Relational Database For a Management System
• Paired Examples as Indirect Supervision in Latent Decision Models
• Model-based Interactive Semantic Parsing: A Unified Framework and A Text-to-SQL Case Study
• Turing: an Accurate and Interpretable Multi-Hypothesis Cross-Domain Natural Language Database Interface
• Question Answering over Knowledge Graphs with Neural Machine Translation and Entity Linking
• How BPE Affects Memorization in Transformers
• Learning Algebraic Recombination for Compositional Generalization
• LAGr: Labeling Aligned Graphs for Improving Systematic Generalization in Semantic Parsing
• End-to-End Entity Resolution and Question Answering Using Differentiable Knowledge Graphs
• Expanding End-to-End Question Answering on Differentiable Knowledge Graphs with Intersection
• COVR: A test-bed for Visually Grounded Compositional Generalization with real images
• Grounded Graph Decoding Improves Compositional Generalization in Question Answering
• Learning to Generalize Compositionally by Transferring Across Semantic Parsing Tasks
• "What Do You Mean by That?" A Parser-Independent Interactive Approach for Enhancing Text-to-SQL
• Revisit Systematic Generalization via Meaningful Learning