Pre-training Text Representations as Meta Learning
arXiv:2004.05568
Abstract
Pre-training text representations has recently been shown to significantly improve the state-of-the-art in many natural language processing tasks. The central goal of pre-training is to learn text representations that are useful for subsequent tasks. However, existing approaches are optimized by minimizing a proxy objective, such as the negative log likelihood of language modeling. In this work, we introduce a learning algorithm which directly optimizes model's ability to learn text representations for effective learning of downstream tasks. We show that there is an intrinsic connection between multi-task pre-training and model-agnostic meta-learning with a sequence of meta-train steps. The standard multi-task learning objective adopted in BERT is a special case of our learning algorithm where the depth of meta-train is zero. We study the problem in two settings: unsupervised pre-training and supervised pre-training with different pre-training objects to verify the generality of our approach.Experimental results show that our algorithm brings improvements and learns better initializations for a variety of downstream tasks.
2 figures, 3 tables
References in corpus (11)
- Adam: A Method for Stochastic Optimization
- Distributed Representations of Words and Phrases and their Compositionality
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Prototypical Networks for Few-shot Learning
- Unified Language Model Pre-training for Natural Language Understanding and Generation
- SQuAD: 100,000+ Questions for Machine Comprehension of Text
- Skip-Thought Vectors
- Multi-Task Deep Neural Networks for Natural Language Understanding
- Discourse-Based Objectives for Fast Unsupervised Sentence Representation Learning
- Large-scale Cloze Test Dataset Created by Teachers