A Comprehensive Exploration on WikiSQL with Table-Aware Word Contextualization

Abstract

We present SQLova, the first Natural-language-to-SQL (NL2SQL) model to achieve human performance in WikiSQL dataset. We revisit and discuss diverse popular methods in NL2SQL literature, take a full advantage of BERT {Devlin et al., 2018) through an effective table contextualization method, and coherently combine them, outperforming the previous state of the art by 8.2% and 2.5% in logical form and execution accuracy, respectively. We particularly note that BERT with a seq2seq decoder leads to a poor performance in the task, indicating the importance of a careful design when using such large pretrained models. We also provide a comprehensive analysis on the dataset and our model, which can be helpful for designing future NL2SQL datsets and models. We especially show that our model's performance is near the upper bound in WikiSQL, where we observe that a large portion of the evaluation errors are due to wrong annotations, and our model is already exceeding human performance by 1.3% in execution accuracy.

arXiv commentKR2ML Workshop at NeurIPS 2019, 11 pages, 4 figures

References in corpus (7)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
• The Natural Language Decathlon: Multitask Learning as Question Answering
• SQLNet: Generating Structured Queries From Natural Language Without Reinforcement Learning
• NSML: A Machine Learning Platform That Enables You to Focus on Your Models
• IncSQL: Training Incremental Text-to-SQL Parsers with Non-Deterministic Oracles
• A Transfer-Learnable Natural Language Interface for Databases

Cited by in corpus (38)

• TAPAS: Weakly Supervised Table Parsing via Pre-training
• Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architectures
• GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing
• X-SQL: reinforce schema representation with context
• Hybrid Ranking Network for Text-to-SQL
• Editing-Based SQL Query Generation for Cross-Domain Context-Dependent Questions
• Content Enhanced BERT-based Text-to-SQL Generation
• TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
• Improving Text-to-SQL with Schema Dependency Learning
• Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representation
• Bridging Textual and Tabular Data for Cross-Domain Text-to-SQL Semantic Parsing
• A Discrete Hard EM Approach for Weakly Supervised Question Answering
• Semantic Evaluation for Text-to-SQL with Distilled Test Suites
• Dynamic Hybrid Relation Network for Cross-Domain Context-Dependent Semantic Parsing
• Schema2QA: High-Quality and Low-Cost Q&A Agents for the Structured Web
• Photon: A Robust Cross-Domain Text-to-SQL System
• Did You Ask a Good Question? A Cross-Domain Question Intention Classification Benchmark for Text-to-SQL
• ColloQL: Robust Cross-Domain Text-to-SQL Over Search Queries
• Improving Entity Linking by Modeling Latent Entity Type Information
• SeqGenSQL -- A Robust Sequence Generation Model for Structured Query Language
• Data Augmentation with Hierarchical SQL-to-Question Generation for Cross-domain Text-to-SQL Parsing
• An Imitation Game for Learning Semantic Parsers from User Interaction
• Generating Semantically Valid Adversarial Questions for TableQA
• Self-supervised Text-to-SQL Learning with Header Alignment Training
• WikiTableT: A Large-Scale Data-to-Text Dataset for Generating Wikipedia Article Sections
• Merging Weak and Active Supervision for Semantic Parsing
• IGSQL: Database Schema Interaction Graph Based Neural Model for Context-Dependent Text-to-SQL Generation
• Generation of complex database queries and API calls from natural language utterances
• A Tale of Two Linkings: Dynamically Gating between Schema Linking and Structural Linking for Text-to-SQL Parsing
• Structured Context and High-Coverage Grammar for Conversational Question Answering over Knowledge Graphs
• Dual Reader-Parser on Hybrid Textual and Tabular Evidence for Open Domain Question Answering
• Case-based Reasoning for Natural Language Queries over Knowledge Bases
• Awakening Latent Grounding from Pretrained Language Models for Semantic Parsing
• Syntactic Question Abstraction and Retrieval for Data-Scarce Semantic Parsing
• Leveraging Table Content for Zero-shot Text-to-SQL with Meta-Learning
• "What Do You Mean by That?" A Parser-Independent Interactive Approach for Enhancing Text-to-SQL
• Model-based Interactive Semantic Parsing: A Unified Framework and A Text-to-SQL Case Study
• A Sketch-Based System for Semantic Parsing