Did You Ask a Good Question? A Cross-Domain Question Intention Classification Benchmark for Text-to-SQL
arXiv:2010.12634
Abstract
Neural models have achieved significant results on the text-to-SQL task, in which most current work assumes all the input questions are legal and generates a SQL query for any input. However, in the real scenario, users can input any text that may not be able to be answered by a SQL query. In this work, we propose TriageSQL, the first cross-domain text-to-SQL question intention classification benchmark that requires models to distinguish four types of unanswerable questions from answerable questions. The baseline RoBERTa model achieves a 60% F1 score on the test set, demonstrating the need for further improvement on this task. Our dataset is available at https://github.com/chatc/TriageSQL.
arXiv comment8 pages, 2 figures
References in corpus (10)
- Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
- Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces
- TabFact: A Large-scale Dataset for Table-based Fact Verification
- Improving Text-to-SQL Evaluation Methodology
- A Comprehensive Exploration on WikiSQL with Table-Aware Word Contextualization
- Hybrid Ranking Network for Text-to-SQL
- CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases
- Towards Ecologically Valid Research on Language User Interfaces
- Type- and Content-Driven Synthesis of SQL Queries from Natural Language
- HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data