Robust Transfer Learning with Pretrained Language Models through Adapters
arXiv:2108.02340
Abstract
Transfer learning with large pretrained transformer-based language models like BERT has become a dominating approach for most NLP tasks. Simply fine-tuning those large language models on downstream tasks or combining it with task-specific pretraining is often not robust. In particular, the performance considerably varies as the random seed changes or the number of pretraining and/or fine-tuning iterations varies, and the fine-tuned model is vulnerable to adversarial attack. We propose a simple yet effective adapter-based approach to mitigate these issues. Specifically, we insert small bottleneck layers (i.e., adapter) within each layer of a pretrained model, then fix the pretrained layers and train the adapter layers on the downstream task data, with (1) task-specific unsupervised pretraining and then (2) task-specific supervised training (e.g., classification, sequence labeling). Our experiments demonstrate that such a training scheme leads to improved stability and adversarial robustness in transfer learning to various downstream tasks.
Accepted to ACL 2021
References in corpus (8)
- BioBERT: a pre-trained biomedical language representation model for biomedical text mining
- Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks
- On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines
- Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Models
- Pre-training via Paraphrasing
- Adv-BERT: BERT is not robust on misspellings! Generating nature adversarial samples on BERT
- BERT-ATTACK: Adversarial Attack Against BERT Using BERT
- On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation