An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models

Abstract

A growing number of state-of-the-art transfer learning methods employ language models pretrained on large generic corpora. In this paper we present a conceptually simple and effective transfer learning approach that addresses the problem of catastrophic forgetting. Specifically, we combine the task-specific optimization function with an auxiliary language model objective, which is adjusted during the training process. This preserves language regularities captured by language models, while enabling sufficient adaptation for solving the target task. Our method does not require pretraining or finetuning separate components of the network and we train our models end-to-end in a single step. We present results on a variety of challenging affective and text classification tasks, surpassing well established transfer learning methods with greater level of complexity.

NAACL 2019

References in corpus (7)

• Scikit-learn: Machine Learning in Python
• Distributed Representations of Words and Phrases and their Compositionality
• Sequence to Sequence Learning with Neural Networks
• Natural Language Processing (almost) from Scratch
• A Structured Self-attentive Sentence Embedding
• Semi-supervised Sequence Learning
• Von Mises-Fisher Loss for Training Sequence to Sequence Models with Continuous Outputs

Cited by in corpus (9)

• Encoder-Agnostic Adaptation for Conditional Language Generation
• Bridging the domain gap in cross-lingual document classification
• Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learning
• Diverse Pretrained Context Encodings Improve Document Translation
• LAMP: Label Augmented Multimodal Pretraining
• Learning Spoken Language Representations with Neural Lattice Language Modeling
• One to Transfer All: A Universal Transfer Framework for Vision Foundation Model with Few Data
• Adaptive Transfer Learning on Graph Neural Networks
• Learning ASR-Robust Contextualized Embeddings for Spoken Language Understanding