KaggleDBQA: Realistic Evaluation of Text-to-SQL Parsers

Abstract

The goal of database question answering is to enable natural language querying of real-life relational databases in diverse application domains. Recently, large-scale datasets such as Spider and WikiSQL facilitated novel modeling techniques for text-to-SQL parsing, improving zero-shot generalization to unseen databases. In this work, we examine the challenges that still prevent these techniques from practical deployment. First, we present KaggleDBQA, a new cross-domain evaluation dataset of real Web databases, with domain-specific data types, original formatting, and unrestricted questions. Second, we re-examine the choice of evaluation tasks for text-to-SQL parsers as applied in real-life settings. Finally, we augment our in-domain evaluation task with database documentation, a naturally occurring source of implicit domain knowledge. We show that KaggleDBQA presents a challenge to state-of-the-art zero-shot parsers but a more realistic evaluation setting and creative use of associated database documentation boosts their accuracy by over 13.2%, doubling their performance.

arXiv commentPublished as a conference paper at ACL-IJCNLP 2021

References in corpus (9)

• Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
• Code Generation as a Dual Task of Code Summarization
• GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing
• Editing-Based SQL Query Generation for Cross-Domain Context-Dependent Questions
• Towards Ecologically Valid Research on Language User Interfaces
• Natural Language to Structured Query Generation via Meta-Learning
• Semantic Evaluation for Text-to-SQL with Distilled Test Suites
• Did You Ask a Good Question? A Cross-Domain Question Intention Classification Benchmark for Text-to-SQL
• ColloQL: Robust Cross-Domain Text-to-SQL Over Search Queries