TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data

Abstract

Recent years have witnessed the burgeoning of pretrained language models (LMs) for text-based natural language (NL) understanding tasks. Such models are typically trained on free-form NL text, hence may not be suitable for tasks like semantic parsing over structured data, which require reasoning over both free-form NL questions and structured tabular data (e.g., database tables). In this paper we present TaBERT, a pretrained LM that jointly learns representations for NL sentences and (semi-)structured tables. TaBERT is trained on a large corpus of 26 million tables and their English contexts. In experiments, neural semantic parsers using TaBERT as feature representation layers achieve new best results on the challenging weakly-supervised semantic parsing benchmark WikiTableQuestions, while performing competitively on the text-to-SQL dataset Spider. Implementation of the model will be available at http://fburl.com/TaBERT .

To Appear at ACL 2020

References in corpus (5)

• Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
• ERNIE: Enhanced Representation through Knowledge Integration
• SQLNet: Generating Structured Queries From Natural Language Without Reinforcement Learning
• Assessing BERT's Syntactic Abilities
• Representing Schema Structure with Graph Neural Networks for Text-to-SQL Parsing

Cited by in corpus (12)

• SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training
• DBTagger: Multi-Task Learning for Keyword Mapping in NLIDBs Using Bi-Directional Recurrent Neural Networks
• SubTab: Subsetting Features of Tabular Data for Self-Supervised Representation Learning
• RPT: Relational Pre-trained Transformer Is Almost All You Need towards Democratizing Data Preparation
• CoTexT: Multi-task Learning with Code-Text Transformer
• TabEAno: Table to Knowledge Graph Entity Annotation
• Relation Extraction from Tables using Artificially Generated Metadata
• Contrastive Mixup: Self- and Semi-Supervised learning for Tabular Domain
• ShadowGNN: Graph Projection Neural Network for Text-to-SQL Parser
• Leveraging Table Content for Zero-shot Text-to-SQL with Meta-Learning
• Making Table Understanding Work in Practice
• A question-answering system for aircraft pilots' documentation