Parameter-Efficient Transfer Learning for NLP

Abstract

Fine-tuning large pre-trained models is an effective transfer mechanism in NLP. However, in the presence of many downstream tasks, fine-tuning is parameter inefficient: an entire new model is required for every task. As an alternative, we propose transfer with adapter modules. Adapter modules yield a compact and extensible model; they add only a few trainable parameters per task, and new tasks can be added without revisiting previous ones. The parameters of the original network remain fixed, yielding a high degree of parameter sharing. To demonstrate adapter's effectiveness, we transfer the recently proposed BERT Transformer model to 26 diverse text classification tasks, including the GLUE benchmark. Adapters attain near state-of-the-art performance, whilst adding only a few parameters per task. On GLUE, we attain within 0.4% of the performance of full fine-tuning, adding only 3.6% parameters per task. By contrast, fine-tuning trains 100% of the parameters per task.

References in corpus (3)

• Distributed Representations of Sentences and Documents
• What makes ImageNet good for transfer learning?
• BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning

Cited by in corpus (19)

• LoRA: Low-Rank Adaptation of Large Language Models
• Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
• How fine can fine-tuning be? Learning efficient language models
• Scalable Transfer Learning with Expert Models
• Rainbow Memory: Continual Learning with a Memory of Diverse Samples
• AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuning
• Combining pre-trained language models and structured knowledge
• RGAT: A Deeper Look into Syntactic Dependency Information for Coreference Resolution
• ProtoTransformer: A Meta-Learning Approach to Providing Student Feedback
• Aggregating From Multiple Target-Shifted Sources
• Contextual Lensing of Universal Sentence Representations
• Legal Transformer Models May Not Always Help
• Exceeding the Limits of Visual-Linguistic Multi-Task Learning
• Improved Customer Transaction Classification using Semi-Supervised Knowledge Distillation
• Parameter-Efficient Methods for Metastases Detection from Clinical Notes
• Cheap Learning: Maximising Performance of Language Models for Social Data Science Using Minimal Data
• Domain Adaptor Networks for Hyperspectral Image Recognition
• Neural Architecture Search for Sentence Classification with BERT
• CERM: Context-aware Literature-based Discovery via Sentiment Analysis