Evaluating the Text-to-SQL Capabilities of Large Language Models

Abstract

We perform an empirical evaluation of Text-to-SQL capabilities of the Codex language model. We find that, without any finetuning, Codex is a strong baseline on the Spider benchmark; we also analyze the failure modes of Codex in this setting. Furthermore, we demonstrate on the GeoQuery and Scholar benchmarks that a small number of in-domain examples provided in the prompt enables Codex to perform better than state-of-the-art models finetuned on such few-shot examples.

Cited by in corpus (4)

• Evaluating Large Language Models in Process Mining: Capabilities, Benchmarks, and Evaluation Strategies
• Metasql: A Generate-then-Rank Framework for Natural Language to SQL Translation
• Mirror: A Natural Language Interface for Data Querying, Summarization, and Visualization
• HiddenTables & PyQTax: A Cooperative Game and Dataset For TableQA to Ensure Scale and Data Privacy Across a Myriad of Taxonomies