The Dawn of Natural Language to SQL: Are We Fully Ready?

Abstract

Translating users' natural language questions into SQL queries (i.e., NL2SQL) significantly lowers the barriers to accessing relational databases. The emergence of Large Language Models has introduced a novel paradigm in NL2SQL tasks, enhancing capabilities dramatically. However, this raises a critical question: Are we fully prepared to deploy NL2SQL models in production? To address the posed questions, we present a multi-angle NL2SQL evaluation framework, NL2SQL360, to facilitate the design and test of new NL2SQL methods for researchers. Through NL2SQL360, we conduct a detailed comparison of leading NL2SQL methods across a range of application scenarios, such as different data domains and SQL characteristics, offering valuable insights for selecting the most appropriate NL2SQL methods for specific needs. Moreover, we explore the NL2SQL design space, leveraging NL2SQL360 to automate the identification of an optimal NL2SQL solution tailored to user-specific needs. Specifically, NL2SQL360 identifies an effective NL2SQL method, SuperSQL, distinguished under the Spdier dataset using the execution accuracy metric. Remarkably, SuperSQL achieves competitive performance with execution accuracy of 87% and 62.66% on the Spider and BIRD test sets, respectively.

VLDB 2024

References in corpus (11)

• Llama 2: Open Foundation and Fine-Tuned Chat Models
• A Survey of Large Language Models
• Code Llama: Open Foundation Models for Code
• Large Language Models: A Survey
• StarCoder: may the source be with you!
• DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligence
• Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs
• Structure-Grounded Pretraining for Text-to-SQL
• C3: Zero-shot Text-to-SQL with ChatGPT
• MAC-SQL: A Multi-Agent Collaborative Framework for Text-to-SQL
• CodeS: Towards Building Open-source Language Models for Text-to-SQL

Cited by in corpus (7)

• A Survey of Text-to-SQL in the Era of LLMs: Where are we, and where are we going?
• Large Language Model for Table Processing: A Survey
• NL2SQL-BUGs: A Benchmark for Detecting Semantic Errors in NL2SQL Translation
• XiYan-SQL: A Novel Multi-Generator Framework For Text-to-SQL
• DeepVIS: Bridging Natural Language and Data Visualization Through Step-wise Reasoning
• SEED: Enhancing Text-to-SQL Performance and Practical Usability Through Automatic Evidence Generation
• SQLMorph: Query Mutation and Fine-Grained Metrics for Text-to-SQL Evaluation