A Comparative Survey of Recent Natural Language Interfaces for Databases

Abstract

Over the last few years natural language interfaces (NLI) for databases have gained significant traction both in academia and industry. These systems use very different approaches as described in recent survey papers. However, these systems have not been systematically compared against a set of benchmark questions in order to rigorously evaluate their functionalities and expressive power. In this paper, we give an overview over 24 recently developed NLIs for databases. Each of the systems is evaluated using a curated list of ten sample questions to show their strengths and weaknesses. We categorize the NLIs into four groups based on the methodology they are using: keyword-, pattern-, parsing-, and grammar-based NLI. Overall, we learned that keyword-based systems are enough to answer simple questions. To solve more complex questions involving subqueries, the system needs to apply some sort of parsing to identify structural dependencies. Grammar-based systems are overall the most powerful ones, but are highly dependent on their manually designed rules. In addition to providing a systematic analysis of the major systems, we derive lessons learned that are vital for designing NLIs that can answer a wide range of user questions.

References in corpus (5)

• Sequence to Sequence Learning with Neural Networks
• A "missing" family of classical orthogonal polynomials
• Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
• KBQA: Learning Question Answering over QA Corpora and Knowledge Bases
• Language to Logical Form with Neural Attention

Cited by in corpus (15)

• Towards Natural Language Interfaces for Data Visualization: A Survey
• Towards Natural Language-Based Visualization Authoring
• CrossData: Leveraging Text-Data Connections for Authoring Data Documents
• Towards Ecologically Valid Research on Language User Interfaces
• Natural Language Interfaces to Data
• Bio-SODA: Enabling Natural Language Question Answering over Knowledge Graphs without Training Data
• PATSQL: Efficient Synthesis of SQL Queries from Example Tables with Quick Inference of Projected Columns
• Spider4SPARQL: A Complex Benchmark for Evaluating Knowledge Graph Question Answering Systems
• DBTagger: Multi-Task Learning for Keyword Mapping in NLIDBs Using Bi-Directional Recurrent Neural Networks
• INODE: Building an End-to-End Data Exploration System in Practice [Extended Vision]
• Using LLMs for Analyzing AIS Data
• A Multilingual Translator to SQL with Database Schema Pruning to Improve Self-Attention
• Towards Enhancing Database Education: Natural Language Generation Meets Query Execution Plans
• mRAT-SQL+GAP:A Portuguese Text-to-SQL Transformer
• ScienceBenchmark: A Complex Real-World Benchmark for Evaluating Natural Language to SQL Systems