TURL: Table Understanding through Representation Learning

Abstract

Relational tables on the Web store a vast amount of knowledge. Owing to the wealth of such tables, there has been tremendous progress on a variety of tasks in the area of table understanding. However, existing work generally relies on heavily-engineered task-specific features and model architectures. In this paper, we present TURL, a novel framework that introduces the pre-training/fine-tuning paradigm to relational Web tables. During pre-training, our framework learns deep contextualized representations on relational tables in an unsupervised manner. Its universal model design with pre-trained representations can be applied to a wide range of tasks with minimal task-specific fine-tuning. Specifically, we propose a structure-aware Transformer encoder to model the row-column structure of relational tables, and present a new Masked Entity Recovery (MER) objective for pre-training to capture the semantics and knowledge in large-scale unlabeled data. We systematically evaluate TURL with a benchmark consisting of 6 different tasks for table understanding (e.g., relation extraction, cell filling). We show that TURL generalizes well to all tasks and substantially outperforms existing methods in almost all instances.

Accepted to VLDB 2021. Extended version with experiments added during revision. Our source code, benchmark, as well as pre-trained models will be available on https://github.com/sunlab-osu/TURL

References in corpus (8)

• Distributed Representations of Words and Phrases and their Compositionality
• TAPAS: Weakly Supervised Table Parsing via Pre-training
• TinyBERT: Distilling BERT for Natural Language Understanding
• Table2Vec: Neural Word and Entity Embeddings for Table Population and Retrieval
• TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
• Leveraging 2-hop Distant Supervision from Table Entity Pairs for Relation Extraction
• Learning Semantic Annotations for Tabular Data
• ColNet: Embedding the Semantics of Web Tables for Column Type Prediction

Cited by in corpus (13)

• Towards Natural Language Interfaces for Data Visualization: A Survey
• TAPEX: Table Pre-training via Learning a Neural SQL Executor
• Annotating Columns with Pre-trained Language Models
• GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing
• GitTables: A Large-Scale Corpus of Relational Tables
• ArcheType: A Novel Framework for Open-Source Column Type Annotation using Large Language Models
• RPT: Relational Pre-trained Transformer Is Almost All You Need towards Democratizing Data Preparation
• A Unified Transferable Model for ML-Enhanced DBMS
• BERT Meets Relational DB: Contextual Representations of Relational Databases
• Relation Extraction from Tables using Artificially Generated Metadata
• TABBIE: Pretrained Representations of Tabular Data
• Making Table Understanding Work in Practice
• Vectorizing string entries for data processing on tables: when are larger language models better?