TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data

Abstract

Recent years have witnessed the burgeoning of pretrained language models (LMs) for text-based natural language (NL) understanding tasks. Such models are typically trained on free-form NL text, hence may not be suitable for tasks like semantic parsing over structured data, which require reasoning over both free-form NL questions and structured tabular data (e.g., database tables). In this paper we present TaBERT, a pretrained LM that jointly learns representations for NL sentences and (semi-)structured tables. TaBERT is trained on a large corpus of 26 million tables and their English contexts. In experiments, neural semantic parsers using TaBERT as feature representation layers achieve new best results on the challenging weakly-supervised semantic parsing benchmark WikiTableQuestions, while performing competitively on the text-to-SQL dataset Spider. Implementation of the model will be available at http://fburl.com/TaBERT .

arXiv commentTo Appear at ACL 2020

References in corpus (14)

• Gaussian Error Linear Units (GELUs)
• Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
• ERNIE: Enhanced Representation through Knowledge Integration
• SQLNet: Generating Structured Queries From Natural Language Without Reinforcement Learning
• Assessing BERT's Syntactic Abilities
• A Comprehensive Exploration on WikiSQL with Table-Aware Word Contextualization
• Memory Augmented Policy Optimization for Program Synthesis and Semantic Parsing
• K-BERT: Enabling Language Representation with Knowledge Graph
• IncSQL: Training Incremental Text-to-SQL Parsers with Non-Deterministic Oracles
• X-SQL: reinforce schema representation with context
• Editing-Based SQL Query Generation for Cross-Domain Context-Dependent Questions
• Representing Schema Structure with Graph Neural Networks for Text-to-SQL Parsing
• Semantics-aware BERT for Language Understanding
• SyntaxSQLNet: Syntax Tree Networks for Complex and Cross-DomainText-to-SQL Task

Cited by in corpus (19)

• Deep Neural Networks and Tabular Data: A Survey
• SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training
• Annotating Columns with Pre-trained Language Models
• TURL: Table Understanding through Representation Learning
• GitTables: A Large-Scale Corpus of Relational Tables
• DBTagger: Multi-Task Learning for Keyword Mapping in NLIDBs Using Bi-Directional Recurrent Neural Networks
• SubTab: Subsetting Features of Tabular Data for Self-Supervised Representation Learning
• RPT: Relational Pre-trained Transformer Is Almost All You Need towards Democratizing Data Preparation
• CoTexT: Multi-task Learning with Code-Text Transformer
• TabEAno: Table to Knowledge Graph Entity Annotation
• Around the GLOBE: Numerical Aggregation Question-Answering on Heterogeneous Genealogical Knowledge Graphs with Deep Neural Networks
• Open Question Answering over Tables and Text
• A Graph Representation of Semi-structured Data for Web Question Answering
• Contrastive Mixup: Self- and Semi-Supervised learning for Tabular Domain
• Relation Extraction from Tables using Artificially Generated Metadata
• ShadowGNN: Graph Projection Neural Network for Text-to-SQL Parser
• Leveraging Table Content for Zero-shot Text-to-SQL with Meta-Learning
• Making Table Understanding Work in Practice
• A question-answering system for aircraft pilots' documentation