Should We Be Pre-training? An Argument for End-task Aware Training as an Alternative
arXiv:2109.07437
Abstract
In most settings of practical concern, machine learning practitioners know in advance what end-task they wish to boost with auxiliary tasks. However, widely used methods for leveraging auxiliary data like pre-training and its continued-pretraining variant are end-task agnostic: they rarely, if ever, exploit knowledge of the target task. We study replacing end-task agnostic continued training of pre-trained language models with end-task aware training of said models. We argue that for sufficiently important end-tasks, the benefits of leveraging auxiliary data in a task-aware fashion can justify forgoing the traditional approach of obtaining generic, end-task agnostic representations as with (continued) pre-training. On three different low-resource NLP tasks from two domains, we demonstrate that multi-tasking the end-task and auxiliary objectives results in significantly better downstream task performance than the widely-used task-agnostic continued pre-training paradigm of Gururangan et al. (2020). We next introduce an online meta-learning algorithm that learns a set of multi-task weights to better balance among our multiple auxiliary objectives, achieving further improvements on end-task performance and data efficiency.
arXiv comment18 pages, 4 figures
References in corpus (19)
- Adam: A Method for Stochastic Optimization
- Distributed Representations of Words and Phrases and their Compositionality
- BioBERT: a pre-trained biomedical language representation model for biomedical text mining
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- Language Models are Few-Shot Learners
- An Overview of Multi-Task Learning in Deep Neural Networks
- XLNet: Generalized Autoregressive Pretraining for Language Understanding
- Skip-Thought Vectors
- On First-Order Meta-Learning Algorithms
- The Natural Language Decathlon: Multitask Learning as Question Answering
- Gradient Surgery for Multi-Task Learning
- Meta-Learning Update Rules for Unsupervised Representation Learning
- Weight Poisoning Attacks on Pre-trained Models
- SciBERT: A Pretrained Language Model for Scientific Text
- When Do You Need Billions of Words of Pretraining Data?
- Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling
- S2ORC: The Semantic Scholar Open Research Corpus
- Auxiliary Task Update Decomposition: The Good, The Bad and The Neutral
- Auxiliary Learning by Implicit Differentiation