Text-to-Text Pre-Training for Data-to-Text Tasks

Abstract

We study the pre-train + fine-tune strategy for data-to-text tasks. Our experiments indicate that text-to-text pre-training in the form of T5, enables simple, end-to-end transformer based models to outperform pipelined neural architectures tailored for data-to-text generation, as well as alternative language model based pre-training techniques such as BERT and GPT-2. Importantly, T5 pre-training leads to better generalization, as evidenced by large improvements on out-of-domain test sets. We hope our work serves as a useful baseline for future research, as transfer learning becomes ever more prevalent for data-to-text tasks.

Accepted to INLG-2020

Cited by in corpus (10)

• GenNI: Human-AI Collaboration for Data-Backed Text Generation
• Automatic Construction of Evaluation Suites for Natural Language Generation Datasets
• SynthBio: A Case Study in Human-AI Collaborative Curation of Text Datasets
• Extracting Summary Knowledge Graphs from Long Documents
• Template Guided Text Generation for Task-Oriented Dialogue
• EASE: Extractive-Abstractive Summarization with Explanations
• AGGGEN: Ordering and Aggregating while Generating
• Data-QuestEval: A Referenceless Metric for Data-to-Text Semantic Evaluation
• TCube: Domain-Agnostic Neural Time-series Narration
• JointGT: Graph-Text Joint Representation Learning for Text Generation from Knowledge Graphs