A Multilingual Translator to SQL with Database Schema Pruning to Improve Self-Attention

Abstract

Long sequences of text are challenging in the context of transformers, due to quadratic memory increase in the self-attention mechanism. As this issue directly affects the translation from natural language to SQL queries (as techniques usually take as input a concatenated text with the question and the database schema), we present techniques that allow long text sequences to be handled by transformers with up to 512 input tokens. We propose a training process with database schema pruning (removal of tables and columns names that are useless for the query of interest). In addition, we used a multilingual approach with the mT5-large model fine-tuned with a data-augmented Spider dataset in four languages simultaneously: English, Portuguese, Spanish, and French. Our proposed technique used the Spider dataset and increased the exact set match accuracy results from 0.718 to 0.736 in a validation dataset (Dev). Source code, evaluations, and checkpoints are available at: \underline{https://github.com/C4AI/gap-text2sql}.

arXiv commentThis preprint has not undergone peer review or any post-submission improvements or corrections. The Version of Record of this article is published in International Journal of Information Technology, and is available online at https://doi.org/10.1007/s41870-023-01342-3 . SharedIt link: https://rdcu.be/dff19

References in corpus (9)

• Longformer: The Long-Document Transformer
• Linformer: Self-Attention with Linear Complexity
• A Comparative Survey of Recent Natural Language Interfaces for Databases
• Multilingual Translation with Extensible Multilingual Pretraining and Finetuning
• Poolingformer: Long Document Modeling with Pooling Attention
• Learning Contextual Representations for Semantic Parsing with Generation-Augmented Pre-Training
• PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models
• S${}^2$SQL: Injecting Syntax to Question-Schema Interaction Graph Encoder for Text-to-SQL Parsers
• MultiSpider: Towards Benchmarking Multilingual Text-to-SQL Semantic Parsing