Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Models

Abstract

In natural language processing, it has been observed recently that generalization could be greatly improved by finetuning a large-scale language model pretrained on a large unlabeled corpus. Despite its recent success and wide adoption, finetuning a large pretrained language model on a downstream task is prone to degenerate performance when there are only a small number of training instances available. In this paper, we introduce a new regularization technique, to which we refer as "mixout", motivated by dropout. Mixout stochastically mixes the parameters of two models. We show that our mixout technique regularizes learning to minimize the deviation from one of the two models and that the strength of regularization adapts along the optimization trajectory. We empirically evaluate the proposed mixout and its variants on finetuning a pretrained language model on downstream tasks. More specifically, we demonstrate that the stability of finetuning and the average accuracy greatly increase when we use the proposed approach to regularize finetuning of BERT on downstream tasks in GLUE.

arXiv commentPublished as a conference paper at ICLR 2020

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Overcoming catastrophic forgetting in neural networks
• Natural Language Processing (almost) from Scratch
• Gaussian Error Linear Units (GELUs)
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• EMNIST: an extension of MNIST to handwritten letters
• Variational Dropout and the Local Reparameterization Trick
• Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks
• On Dropout and Nuclear Norm Regularization

Cited by in corpus (10)

• Entailment as Few-Shot Learner
• Incorporating BERT into Parallel Sequence Decoding with Adapters
• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing
• Localising In-Domain Adaptation of Transformer-Based Biomedical Language Models
• Improving Pre-trained Language Model Fine-tuning with Noise Stability Regularization
• Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting
• AF Adapter: Continual Pretraining for Building Chinese Biomedical Language Model
• Robust Transfer Learning with Pretrained Language Models through Adapters
• Joint Text and Label Generation for Spoken Language Understanding
• Renofeation: A Simple Transfer Learning Method for Improved Adversarial Robustness