CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases

Abstract

We present CoSQL, a corpus for building cross-domain, general-purpose database (DB) querying dialogue systems. It consists of 30k+ turns plus 10k+ annotated SQL queries, obtained from a Wizard-of-Oz (WOZ) collection of 3k dialogues querying 200 complex DBs spanning 138 domains. Each dialogue simulates a real-world DB query scenario with a crowd worker as a user exploring the DB and a SQL expert retrieving answers with SQL, clarifying ambiguous questions, or otherwise informing of unanswerable questions. When user questions are answerable by SQL, the expert describes the SQL and execution results to the user, hence maintaining a natural interaction flow. CoSQL introduces new challenges compared to existing task-oriented dialogue datasets:(1) the dialogue states are grounded in SQL, a domain-independent executable representation, instead of domain-specific slot-value pairs, and (2) because testing is done on unseen databases, success requires generalizing to new domains. CoSQL includes three tasks: SQL-grounded dialogue state tracking, response generation from query results, and user dialogue act prediction. We evaluate a set of strong baselines for each task and show that CoSQL presents significant challenges for future research. The dataset, baselines, and leaderboard will be released at https://yale-lily.github.io/cosql.

arXiv commentAccepted to EMNLP 2019, long paper

References in corpus (11)

• Sequence to Sequence Learning with Neural Networks
• Learning to Map Sentences to Logical Form: Structured Classification with Probabilistic Categorial Grammars
• Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
• Deep Reinforcement Learning for Dialogue Generation
• SQLNet: Generating Structured Queries From Natural Language Without Reinforcement Learning
• Improving Text-to-SQL Evaluation Methodology
• A Network-based End-to-End Trainable Task-oriented Dialogue System
• On-line Active Reward Learning for Policy Optimisation in Spoken Dialogue Systems
• IncSQL: Training Incremental Text-to-SQL Parsers with Non-Deterministic Oracles
• Editing-Based SQL Query Generation for Cross-Domain Context-Dependent Questions
• Representing Schema Structure with Graph Neural Networks for Text-to-SQL Parsing

Cited by in corpus (15)

• Editing-Based SQL Query Generation for Cross-Domain Context-Dependent Questions
• Recent Advances and Challenges in Task-oriented Dialog System
• Improving Text-to-SQL with Schema Dependency Learning
• Semantic Evaluation for Text-to-SQL with Distilled Test Suites
• Did You Ask a Good Question? A Cross-Domain Question Intention Classification Benchmark for Text-to-SQL
• Grounded Adaptation for Zero-shot Executable Semantic Parsing
• Unsupervised Learning of KB Queries in Task-Oriented Dialogs
• Decoupled Dialogue Modeling and Semantic Parsing for Multi-Turn Text-to-SQL
• Learning Adaptive Language Interfaces through Decomposition
• Tracking Interaction States for Multi-Turn Text-to-SQL Semantic Parsing
• HyKnow: End-to-End Task-Oriented Dialog Modeling with Hybrid Knowledge Management
• IGSQL: Database Schema Interaction Graph Based Neural Model for Context-Dependent Text-to-SQL Generation
• Dual Reader-Parser on Hybrid Textual and Tabular Evidence for Open Domain Question Answering
• Mapping Language to Programs using Multiple Reward Components with Inverse Reinforcement Learning
• Context Dependent Semantic Parsing: A Survey