Annotating Columns with Pre-trained Language Models

Abstract

Inferring meta information about tables, such as column headers or relationships between columns, is an active research topic in data management as we find many tables are missing some of this information. In this paper, we study the problem of annotating table columns (i.e., predicting column types and the relationships between columns) using only information from the table itself. We develop a multi-task learning framework (called Doduo) based on pre-trained language models, which takes the entire table as input and predicts column types/relations using a single model. Experimental results show that Doduo establishes new state-of-the-art performance on two benchmarks for the column type prediction and column relation prediction tasks with up to 4.0% and 11.9% improvements, respectively. We report that Doduo can already outperform the previous state-of-the-art performance with a minimal number of tokens, only 8 tokens per column. We release a toolbox (https://github.com/megagonlabs/doduo) and confirm the effectiveness of Doduo on a real-world data science problem through a case study.

SIGMOD 2022

References in corpus (13)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• An Overview of Multi-Task Learning in Deep Neural Networks
• Longformer: The Long-Document Transformer
• TAPAS: Weakly Supervised Table Parsing via Pre-training
• Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Case
• Deep Entity Matching with Pre-Trained Language Models
• TUTA: Tree-based Transformers for Generally Structured Table Pre-training
• Annotating Columns with Pre-trained Language Models
• TURL: Table Understanding through Representation Learning
• TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data
• TCN: Table Convolutional Network for Web Table Interpretation
• Semantic Labeling Using a Deep Contextualized Language Model
• Semantic Annotation for Tabular Data

Cited by in corpus (4)

• Towards Natural Language Interfaces for Data Visualization: A Survey
• Annotating Columns with Pre-trained Language Models
• Graph Neural Network Approach to Semantic Type Detection in Tables
• PLM4NDV: Minimizing Data Access for Number of Distinct Values Estimation with Pre-trained Language Models