ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning
arXiv:2111.10952
Abstract
Despite the recent success of multi-task learning and transfer learning for natural language processing (NLP), few works have systematically studied the effect of scaling up the number of tasks during pre-training. Towards this goal, this paper introduces ExMix (Extreme Mixture): a massive collection of 107 supervised NLP tasks across diverse domains and task-families. Using ExMix, we study the effect of multi-task pre-training at the largest scale to date, and analyze co-training transfer amongst common families of tasks. Through this analysis, we show that manually curating an ideal set of tasks for multi-task pre-training is not straightforward, and that multi-task scaling can vastly improve models on its own. Finally, we propose ExT5: a model pre-trained using a multi-task objective of self-supervised span denoising and supervised ExMix. Via extensive experiments, we show that ExT5 outperforms strong T5 baselines on SuperGLUE, GEM, Rainbow, Closed-Book QA tasks, and several tasks outside of ExMix. ExT5 also significantly improves sample efficiency while pre-training.
ICLR 2022; see https://youtu.be/FbRcbM4T-50 for a video overview of the paper
References in corpus (14)
- Scaling Laws for Neural Language Models
- BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
- An Empirical Model of Large-Batch Training
- Pre-training Tasks for Embedding-based Large-scale Retrieval
- Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models
- Finetuned Language Models Are Zero-Shot Learners
- Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
- Unlocking Compositional Generalization in Pre-trained Models Using Intermediate Representations
- On Better Exploring and Exploiting Task Relationships in Multi-Task Learning: Joint Model and Feature Learning
- Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets
- Efficiently Identifying Task Groupings for Multi-Task Learning
- GLU Variants Improve Transformer
- When Do You Need Billions of Words of Pretraining Data?
- AgreeSum: Agreement-Oriented Multi-Document Summarization