On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation

Abstract

Adapter-based tuning has recently arisen as an alternative to fine-tuning. It works by adding light-weight adapter modules to a pretrained language model (PrLM) and only updating the parameters of adapter modules when learning on a downstream task. As such, it adds only a few trainable parameters per new task, allowing a high degree of parameter sharing. Prior studies have shown that adapter-based tuning often achieves comparable results to fine-tuning. However, existing work only focuses on the parameter-efficient aspect of adapter-based tuning while lacking further investigation on its effectiveness. In this paper, we study the latter. We first show that adapter-based tuning better mitigates forgetting issues than fine-tuning since it yields representations with less deviation from those generated by the initial PrLM. We then empirically compare the two tuning methods on several downstream NLP tasks and settings. We demonstrate that 1) adapter-based tuning outperforms fine-tuning on low-resource and cross-lingual tasks; 2) it is more robust to overfitting and less sensitive to changes in learning rates.

arXiv commentAccepted by ACL 2021 (long paper)

References in corpus (9)

• FreeLB: Enhanced Adversarial Training for Natural Language Understanding
• Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks
• Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping
• BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning
• How Transferable are Neural Networks in NLP Applications?
• Correlating neural and symbolic representations of language
• Visualizing and Understanding the Effectiveness of BERT
• Simple, Scalable Adaptation for Neural Machine Translation
• Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting

Cited by in corpus (5)

• Position is Power: System Prompts as a Mechanism of Bias in Large Language Models (LLMs)
• Semi-Siamese Bi-encoder Neural Ranking Model Using Lightweight Fine-Tuning
• Unsupervised Domain Adaptation with Adapter
• AdapterEM: Pre-trained Language Model Adaptation for Generalized Entity Matching using Adapter-tuning
• Robust Transfer Learning with Pretrained Language Models through Adapters