Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning

Abstract

A significant amount of the world's knowledge is stored in relational databases. However, the ability for users to retrieve facts from a database is limited due to a lack of understanding of query languages such as SQL. We propose Seq2SQL, a deep neural network for translating natural language questions to corresponding SQL queries. Our model leverages the structure of SQL queries to significantly reduce the output space of generated queries. Moreover, we use rewards from in-the-loop query execution over the database to learn a policy to generate unordered parts of the query, which we show are less suitable for optimization via cross entropy loss. In addition, we will publish WikiSQL, a dataset of 80654 hand-annotated examples of questions and SQL queries distributed across 24241 tables from Wikipedia. This dataset is required to train our model and is an order of magnitude larger than comparable datasets. By applying policy-based reinforcement learning with a query execution environment to WikiSQL, our model Seq2SQL outperforms attentional sequence to sequence models, improving execution accuracy from 35.9% to 59.4% and logical form accuracy from 23.4% to 48.3%.

12 pages, 5 figures

References in corpus (5)

• Learning to Map Sentences to Logical Form: Structured Classification with Probabilistic Categorial Grammars
• Neural Combinatorial Optimization with Reinforcement Learning
• Compositional Semantic Parsing on Semi-Structured Tables
• From Language to Programs: Bridging Reinforcement Learning and Maximum Marginal Likelihood
• Natural Language Interfaces to Databases - An Introduction

Cited by in corpus (200)

• LoRA: Low-Rank Adaptation of Large Language Models
• A Survey of Human-in-the-loop for Machine Learning
• TAPAS: Weakly Supervised Table Parsing via Pre-training
• Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming
• The Natural Language Decathlon: Multitask Learning as Question Answering
• SQLNet: Generating Structured Queries From Natural Language Without Reinforcement Learning
• TabFact: A Large-scale Dataset for Table-based Fact Verification
• Improving Text-to-SQL Evaluation Methodology
• NL4DV: A Toolkit for Generating Analytic Specifications for Data Visualization from Natural Language Queries
• Towards Natural Language Interfaces for Data Visualization: A Survey
• A Comparative Survey of Recent Natural Language Interfaces for Databases
• QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension
• Graph2Seq: Graph to Sequence Learning with Attention-based Neural Networks
• Building a Conversational Agent Overnight with Dialogue Self-Play
• FlowSense: A Natural Language Interface for Visual Data Exploration within a Dataflow System
• Database Meets Deep Learning: Challenges and Opportunities
• Natural Language Generation and Understanding of Big Code for AI-Assisted Programming: A Review
• A Comprehensive Exploration on WikiSQL with Table-Aware Word Contextualization
• LAMOL: LAnguage MOdeling for Lifelong Language Learning
• TAPEX: Table Pre-training via Learning a Neural SQL Executor
• Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architectures
• Towards Natural Language-Based Visualization Authoring
• Graph Neural Networks for Natural Language Processing: A Survey
• Learning to Generalize from Sparse and Underspecified Rewards
• GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing
• A Survey of Deep Learning for Scientific Discovery
• IncSQL: Training Incremental Text-to-SQL Parsers with Non-Deterministic Oracles
• X-SQL: reinforce schema representation with context
• Hybrid Ranking Network for Text-to-SQL
• Editing-Based SQL Query Generation for Cross-Domain Context-Dependent Questions
• Content Enhanced BERT-based Text-to-SQL Generation
• Grammar-based Neural Text-to-SQL Generation
• CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases
• CrossData: Leveraging Text-Data Connections for Authoring Data Documents
• Seq2Slate: Re-ranking and Slate Optimization with RNNs
• Coarse-to-Fine Decoding for Neural Semantic Parsing
• Introduction to Neural Network based Approaches for Question Answering over Knowledge Graphs
• Representing Schema Structure with Graph Neural Networks for Text-to-SQL Parsing
• Natural Language Interfaces to Data
• TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
• Improving Text-to-SQL with Schema Dependency Learning
• Can We Generate Shellcodes via Natural Language? An Empirical Study
• SyntaxSQLNet: Syntax Tree Networks for Complex and Cross-DomainText-to-SQL Task
• A Survey on Semantic Parsing
• Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representation
• SpatialNLI: A Spatial Domain Natural Language Interface to Databases Using Spatial Comprehension
• TRANX: A Transition-based Neural Abstract Syntax Parser for Semantic Parsing and Code Generation
• A Transfer-Learnable Natural Language Interface for Databases
• CraftAssist: A Framework for Dialogue-enabled Interactive Agents
• A View on Deep Reinforcement Learning in System Optimization
• Hierarchical Poset Decoding for Compositional Generalization in Language
• Neural Machine Translation for Query Construction and Composition
• Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task
• Scalable Neural Methods for Reasoning With a Symbolic Knowledge Base
• Adversarial TableQA: Attention Supervision for Question Answering on Tables
• Bridging Textual and Tabular Data for Cross-Domain Text-to-SQL Semantic Parsing
• A Discrete Hard EM Approach for Weakly Supervised Question Answering
• Recent Advances in Neural Program Synthesis
• TableQA: a Large-Scale Chinese Text-to-SQL Dataset for Table-Aware SQL Generation
• Knowledge Graph-based Question Answering with Electronic Health Records
• Towards Robustness of Text-to-SQL Models against Synonym Substitution
• Neural Machine Translating from Natural Language to SPARQL
• Semantic Parsing with Syntax- and Table-Aware SQL Generation
• Quda: Natural Language Queries for Visual Data Analytics
• One-Shot Learning for Text-to-SQL Generation
• HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data
• Semantic Evaluation for Text-to-SQL with Distilled Test Suites
• Spider4SPARQL: A Complex Benchmark for Evaluating Knowledge Graph Question Answering Systems
• Building a Neural Semantic Parser from a Domain Ontology
• Text-to-OverpassQL: A Natural Language Interface for Complex Geodata Querying of OpenStreetMap
• GP: Context-free Grammar Pre-training for Text-to-SQL Parsers
• Dynamic Hybrid Relation Network for Cross-Domain Context-Dependent Semantic Parsing
• SPoC: Search-based Pseudocode to Code
• Coupling Retrieval and Meta-Learning for Context-Dependent Semantic Parsing
• DBTagger: Multi-Task Learning for Keyword Mapping in NLIDBs Using Bi-Directional Recurrent Neural Networks
• Why Build an Assistant in Minecraft?
• Photon: A Robust Cross-Domain Text-to-SQL System
• LGESQL: Line Graph Enhanced Text-to-SQL Model with Mixed Local and Non-Local Relations
• Encoding Database Schemas with Relation-Aware Self-Attention for Text-to-SQL Parsers
• TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance
• Bidirectional Attention for SQL Generation
• Toward Code Generation: A Survey and Lessons from Semantic Parsing
• Text-to-SQL Generation for Question Answering on Electronic Medical Records
• Did You Ask a Good Question? A Cross-Domain Question Intention Classification Benchmark for Text-to-SQL
• MultiTabQA: Generating Tabular Answers for Multi-Table Question Answering
• Frameworks for Querying Databases Using Natural Language: A Literature Review
• CraftAssist Instruction Parsing: Semantic Parsing for a Minecraft Assistant
• Text-to-SQL in the Wild: A Naturally-Occurring Dataset Based on Stack Exchange Data
• Mirror: A Natural Language Interface for Data Querying, Summarization, and Visualization
• ColloQL: Robust Cross-Domain Text-to-SQL Over Search Queries
• Compositional Generalization via Semantic Tagging
• Exploring Neural Models for Parsing Natural Language into First-Order Logic
• JuICe: A Large Scale Distantly Supervised Dataset for Open Domain Context-based Code Generation
• Lyra: A Benchmark for Turducken-Style Code Generation
• Attention-based Aspect Reasoning for Knowledge Base Question Answering on Clinical Notes
• Set-to-Sequence Methods in Machine Learning: a Review
• Grounded Adaptation for Zero-shot Executable Semantic Parsing
• Neural Semantic Parsing in Low-Resource Settings with Back-Translation and Meta-Learning
• SQL-to-Text Generation with Graph-to-Sequence Model
• Lifelong Language Knowledge Distillation
• Towards Natural Language Question Answering over Earth Observation Linked Data using Attention-based Neural Machine Translation
• Optimal Completion Distillation for Sequence Learning
• Energy-Based Models for Code Generation under Compilability Constraints
• Improving the Capabilities of Large Language Model Based Marketing Analytics Copilots With Semantic Search And Fine-Tuning
• Table2answer: Read the database and answer without SQL
• Jointly Learning Semantic Parser and Natural Language Generator via Dual Information Maximization
• Around the GLOBE: Numerical Aggregation Question-Answering on Heterogeneous Genealogical Knowledge Graphs with Deep Neural Networks
• SeqGenSQL -- A Robust Sequence Generation Model for Structured Query Language
• Learning Semantic Parsers from Denotations with Latent Structured Alignments and Abstract Programs
• Query Understanding for Natural Language Enterprise Search
• FeTaQA: Free-form Table Question Answering
• AutoQA: From Databases To QA Semantic Parsers With Only Synthetic Training Data
• Question Generation from SQL Queries Improves Neural Semantic Parsing
• Interactive Semantic Parsing for If-Then Recipes via Hierarchical Reinforcement Learning
• Building an Application Independent Natural Language Interface
• Meta-Learning for Domain Generalization in Semantic Parsing
• Open Question Answering over Tables and Text
• An Imitation Game for Learning Semantic Parsers from User Interaction
• Data Augmentation with Hierarchical SQL-to-Question Generation for Cross-domain Text-to-SQL Parsing
• Decoupled Dialogue Modeling and Semantic Parsing for Multi-Turn Text-to-SQL
• Policy Shaping and Generalized Update Equations for Semantic Parsing from Denotations
• TabGenie: A Toolkit for Table-to-Text Generation
• Zero-shot Text-to-SQL Learning with Auxiliary Task
• Zero-shot Transfer Learning for Semantic Parsing
• Generating Semantically Valid Adversarial Questions for TableQA
• Using Database Rule for Weak Supervised Text-to-SQL Generation
• A Hybrid Semantic Parsing Approach for Tabular Data Analysis
• Learning an Executable Neural Semantic Parser
• CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP
• Subjective Question Answering: Deciphering the inner workings of Transformers in the realm of subjectivity
• HiddenTables & PyQTax: A Cooperative Game and Dataset For TableQA to Ensure Scale and Data Privacy Across a Myriad of Taxonomies
• Towards Enhancing Database Education: Natural Language Generation Meets Query Execution Plans
• Scrutinizer: A Mixed-Initiative Approach to Large-Scale, Data-Driven Claim Verification
• Adaptive Correlated Monte Carlo for Contextual Categorical Sequence Generation
• Neural-Symbolic Solver for Math Word Problems with Auxiliary Tasks
• Exploring Underexplored Limitations of Cross-Domain Text-to-SQL Generalization
• An Investigation Between Schema Linking and Text-to-SQL Performance
• Self-supervised Text-to-SQL Learning with Header Alignment Training
• A Pilot Study for Chinese SQL Semantic Parsing
• Prefix-to-SQL: Text-to-SQL Generation from Incomplete User Questions
• CLTR: An End-to-End, Transformer-Based System for Cell Level Table Retrieval and Table Question Answering
• Translating Natural Language to SQL using Pointer-Generator Networks and How Decoding Order Matters
• Domain Adaptation for Semantic Parsing
• End-to-End Cross-Domain Text-to-SQL Semantic Parsing with Auxiliary Task
• Context-Dependent Semantic Parsing over Temporally Structured Data
• Questions to Guide the Future of Artificial Intelligence Research
• Transferable Natural Language Interface to Structured Queries aided by Adversarial Generation
• IGSQL: Database Schema Interaction Graph Based Neural Model for Context-Dependent Text-to-SQL Generation
• MLFriend: Interactive Prediction Task Recommendation for Event-Driven Time-Series Data
• Tracking Interaction States for Multi-Turn Text-to-SQL Semantic Parsing
• mRAT-SQL+GAP:A Portuguese Text-to-SQL Transformer
• NeurIPS 2020 NLC2CMD Competition: Translating Natural Language to Bash Commands
• Program Enhanced Fact Verification with Verbalization and Graph Attention Network
• MT-Teql: Evaluating and Augmenting Consistency of Text-to-SQL Models with Metamorphic Testing
• Code Generation from Natural Language with Less Prior and More Monolingual Data
• Question Answering via Web Extracted Tables and Pipelined Models
• Sigma: A dataset for text-to-code semantic parsing with statistical analysis
• SEED: Enhancing Text-to-SQL Performance and Practical Usability Through Automatic Evidence Generation
• ShadowGNN: Graph Projection Neural Network for Text-to-SQL Parser
• Can Deep Neural Networks Predict Data Correlations from Column Names?
• Siamese Networks for Semantic Pattern Similarity
• Merging Weak and Active Supervision for Semantic Parsing
• Turing: an Accurate and Interpretable Multi-Hypothesis Cross-Domain Natural Language Database Interface
• SPARQLing Database Queries from Intermediate Question Decompositions
• Differentiable Representations For Multihop Inference Rules
• Dual Reader-Parser on Hybrid Textual and Tabular Evidence for Open Domain Question Answering
• Multilingual Neural Semantic Parsing for Low-Resourced Languages
• On Adversarial Robustness of Synthetic Code Generation
• Doc2Dict: Information Extraction as Text Generation
• KaggleDBQA: Realistic Evaluation of Text-to-SQL Parsers
• Case-based Reasoning for Natural Language Queries over Knowledge Bases
• Sattiy at SemEval-2021 Task 9: An Ensemble Solution for Statement Verification and Evidence Finding with Tables
• Semantic Parsing Natural Language into Relational Algebra
• Multi-modal Retrieval of Tables and Texts Using Tri-encoder Models
• Natural SQL: Making SQL Easier to Infer from Natural Language Specifications
• Leveraging Table Content for Zero-shot Text-to-SQL with Meta-Learning
• Keeping Notes: Conditional Natural Language Generation with a Scratchpad Mechanism
• A Tale of Two Linkings: Dynamically Gating between Schema Linking and Structural Linking for Text-to-SQL Parsing
• Mapping Language to Programs using Multiple Reward Components with Inverse Reinforcement Learning
• Learning Representations for Zero-Shot Retrieval over Structured Data
• Reducing the impact of out of vocabulary words in the translation of natural language questions into SPARQL queries
• MK-SQuIT: Synthesizing Questions using Iterative Template-filling
• "What Do You Mean by That?" A Parser-Independent Interactive Approach for Enhancing Text-to-SQL
• Generation of complex database queries and API calls from natural language utterances
• Understanding in Artificial Intelligence
• Distantly Supervised Question Parsing
• Metaknowledge Extraction Based on Multi-Modal Documents
• Answering Conversational Questions on Structured Data without Logical Forms
• Semantic Parsing to Manipulate Relational Database For a Management System
• LogicalFactChecker: Leveraging Logical Operations for Fact Checking with Graph Module Network
• Syntactic Question Abstraction and Retrieval for Data-Scarce Semantic Parsing
• TAG : Type Auxiliary Guiding for Code Comment Generation
• End-to-End Learning for Answering Structured Queries Directly over Text
• Insightful Assistant: AI-compatible Operation Graph Representations for Enhancing Industrial Conversational Agents
• Model-based Interactive Semantic Parsing: A Unified Framework and A Text-to-SQL Case Study
• Mapping Natural Language Commands to Web Elements
• Look-up and Adapt: A One-shot Semantic Parser
• Sequence to Logic with Copy and Cache
• Pragmatic approach to structured data querying via natural language interface
• FANDA: A Novel Approach to Perform Follow-up Query Analysis