Robustly Optimized and Distilled Training for Natural Language Understanding
arXiv:2103.08809
Abstract
In this paper, we explore multi-task learning (MTL) as a second pretraining step to learn enhanced universal language representation for transformer language models. We use the MTL enhanced representation across several natural language understanding tasks to improve performance and generalization. Moreover, we incorporate knowledge distillation (KD) in MTL to further boost performance and devise a KD variant that learns effectively from multiple teachers. By combining MTL and KD, we propose Robustly Optimized and Distilled (ROaD) modeling framework. We use ROaD together with the ELECTRA model to obtain state-of-the-art results for machine reading comprehension and natural language inference.
References in corpus (7)
- Distilling the Knowledge in a Neural Network
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- Teaching Machines to Read and Comprehend
- A Survey on Multi-Task Learning
- DeBERTa: Decoding-enhanced BERT with Disentangled Attention
- Multi-Task Deep Neural Networks for Natural Language Understanding
- NewsQA: A Machine Comprehension Dataset