Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling

Abstract

Natural language understanding has recently seen a surge of progress with the use of sentence encoders like ELMo (Peters et al., 2018a) and BERT (Devlin et al., 2019) which are pretrained on variants of language modeling. We conduct the first large-scale systematic study of candidate pretraining tasks, comparing 19 different tasks both as alternatives and complements to language modeling. Our primary results support the use language modeling, especially when combined with pretraining on additional labeled-data tasks. However, our results are mixed across pretraining tasks and show some concerning trends: In ELMo's pretrain-then-freeze paradigm, random baselines are worryingly strong and results vary strikingly across target tasks. In addition, fine-tuning BERT on an intermediate task often negatively impacts downstream transfer. In a more positive trend, we see modest gains from multitask training, suggesting the development of more sophisticated multitask and transfer learning techniques as an avenue for further research.

arXiv commentACL 2019. This paper supercedes "Looking for ELMo's Friends: Sentence-Level Pretraining Beyond Language Modeling", an earlier version of this work by the same authors

References in corpus (14)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Natural Language Processing (almost) from Scratch
• On the Convergence of Adam and Beyond
• Bidirectional Attention Flow for Machine Comprehension
• Skip-Thought Vectors
• Achieving Human Parity on Automatic Chinese to English News Translation
• Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks
• Learning and Evaluating General Linguistic Intelligence
• BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning
• Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning
• No Training Required: Exploring Random Encoders for Sentence Classification
• Multi-Task Learning for Sequence Tagging: An Empirical Study
• DisSent: Sentence Representation Learning from Explicit Discourse Relations
• Language Modeling Teaches You More Syntax than Translation Does: Lessons Learned Through Auxiliary Task Analysis

Cited by in corpus (3)

• Should We Be Pre-training? An Argument for End-task Aware Training as an Alternative
• Predicting the Argumenthood of English Prepositional Phrases
• Multi-level Head-wise Match and Aggregation in Transformer for Textual Sequence Matching