Semantic Evaluation for Text-to-SQL with Distilled Test Suites

Abstract

We propose test suite accuracy to approximate semantic accuracy for Text-to-SQL models. Our method distills a small test suite of databases that achieves high code coverage for the gold query from a large number of randomly generated databases. At evaluation time, it computes the denotation accuracy of the predicted queries on the distilled test suite, hence calculating a tight upper-bound for semantic accuracy efficiently. We use our proposed method to evaluate 21 models submitted to the Spider leader board and manually verify that our method is always correct on 100 examples. In contrast, the current Spider metric leads to a 2.5% false negative rate on average and 8.1% in the worst case, indicating that test suite accuracy is needed. Our implementation, along with distilled test suites for eleven Text-to-SQL datasets, is publicly available.

EMNLP 2020 Long Paper

References in corpus (12)

• Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
• The Natural Language Decathlon: Multitask Learning as Question Answering
• SQLNet: Generating Structured Queries From Natural Language Without Reinforcement Learning
• Improving Text-to-SQL Evaluation Methodology
• Lambda Dependency-Based Compositional Semantics
• A Comprehensive Exploration on WikiSQL with Table-Aware Word Contextualization
• Memory Augmented Policy Optimization for Program Synthesis and Semantic Parsing
• X-SQL: reinforce schema representation with context
• Hybrid Ranking Network for Text-to-SQL
• CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases
• Representing Schema Structure with Graph Neural Networks for Text-to-SQL Parsing
• Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representation

Cited by in corpus (5)

• MT-Teql: Evaluating and Augmenting Consistency of Text-to-SQL Models with Metamorphic Testing
• mRAT-SQL+GAP:A Portuguese Text-to-SQL Transformer
• NeurIPS 2020 NLC2CMD Competition: Translating Natural Language to Bash Commands
• KaggleDBQA: Realistic Evaluation of Text-to-SQL Parsers
• LAGr: Labeling Aligned Graphs for Improving Systematic Generalization in Semantic Parsing