Self-supervised Text-to-SQL Learning with Header Alignment Training

Abstract

Since we can leverage a large amount of unlabeled data without any human supervision to train a model and transfer the knowledge to target tasks, self-supervised learning is a de-facto component for the recent success of deep learning in various fields. However, in many cases, there is a discrepancy between a self-supervised learning objective and a task-specific objective. In order to tackle such discrepancy in Text-to-SQL task, we propose a novel self-supervised learning framework. We utilize the task-specific properties of Text-to-SQL task and the underlying structures of table contents to train the models to learn useful knowledge of the \textit{header-column} alignment task from unlabeled table data. We are able to transfer the knowledge to the supervised Text-to-SQL training with annotated samples, so that the model can leverage the knowledge to better perform the \textit{header-span} alignment task to predict SQL statements. Experimental results show that our self-supervised learning framework significantly improves the performance of the existing strong BERT based models without using large external corpora. In particular, our method is effective for training the model with scarce labeled data. The source code of this work is available in GitHub.

References in corpus (13)

• A Simple Framework for Contrastive Learning of Visual Representations
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
• Learning to Map Sentences to Logical Form: Structured Classification with Probabilistic Categorial Grammars
• Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
• A Comprehensive Exploration on WikiSQL with Table-Aware Word Contextualization
• Language Models as Knowledge Bases?
• Memory Augmented Policy Optimization for Program Synthesis and Semantic Parsing
• Learning to Generalize from Sparse and Underspecified Rewards
• IncSQL: Training Incremental Text-to-SQL Parsers with Non-Deterministic Oracles
• Grammar-based Neural Text-to-SQL Generation
• SyntaxSQLNet: Syntax Tree Networks for Complex and Cross-DomainText-to-SQL Task
• Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representation