Parameter-Efficient Transfer Learning for NLP

Abstract

Fine-tuning large pre-trained models is an effective transfer mechanism in NLP. However, in the presence of many downstream tasks, fine-tuning is parameter inefficient: an entire new model is required for every task. As an alternative, we propose transfer with adapter modules. Adapter modules yield a compact and extensible model; they add only a few trainable parameters per task, and new tasks can be added without revisiting previous ones. The parameters of the original network remain fixed, yielding a high degree of parameter sharing. To demonstrate adapter's effectiveness, we transfer the recently proposed BERT Transformer model to 26 diverse text classification tasks, including the GLUE benchmark. Adapters attain near state-of-the-art performance, whilst adding only a few parameters per task. On GLUE, we attain within 0.4% of the performance of full fine-tuning, adding only 3.6% parameters per task. By contrast, fine-tuning trains 100% of the parameters per task.

References in corpus (13)

• Distributed Representations of Words and Phrases and their Compositionality
• Overcoming catastrophic forgetting in neural networks
• Distributed Representations of Sentences and Documents
• In Defense of the Triplet Loss for Person Re-Identification
• Universal Sentence Encoder
• Continual Learning Through Synaptic Intelligence
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• Semi-supervised Sequence Learning
• What makes ImageNet good for transfer learning?
• Learned in Translation: Contextualized Word Vectors
• BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning
• On Self Modulation for Generative Adversarial Networks
• Transfer Learning with Neural AutoML

Cited by in corpus (39)

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• LoRA: Low-Rank Adaptation of Large Language Models
• Pre-trained Models for Natural Language Processing: A Survey
• Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
• Revisiting Few-sample BERT Fine-tuning
• Compacter: Efficient Low-Rank Hypercomplex Adapter Layers
• A Survey of Deep Learning for Scientific Discovery
• Incorporating BERT into Parallel Sequence Decoding with Adapters
• Task-Oriented Dialogue System as Natural Language Generation
• Multilingual Speech Translation with Efficient Finetuning of Pretrained Models
• How fine can fine-tuning be? Learning efficient language models
• Scalable Transfer Learning with Expert Models
• Encoder-Agnostic Adaptation for Conditional Language Generation
• CATBERT: Context-Aware Tiny BERT for Detecting Social Engineering Emails
• Rainbow Memory: Continual Learning with a Memory of Diverse Samples
• AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuning
• Simple, Scalable Adaptation for Neural Machine Translation
• Combining pre-trained language models and structured knowledge
• RGAT: A Deeper Look into Syntactic Dependency Information for Coreference Resolution
• Parameter-Efficient Transfer from Sequential Behaviors for User Modeling and Recommendation
• Does Non-COVID19 Lung Lesion Help? Investigating Transferability in COVID-19 CT Image Segmentation
• ProtoTransformer: A Meta-Learning Approach to Providing Student Feedback
• Aggregating From Multiple Target-Shifted Sources
• On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment
• Contextual Lensing of Universal Sentence Representations
• Legal Transformer Models May Not Always Help
• Style Attuned Pre-training and Parameter Efficient Fine-tuning for Spoken Language Understanding
• Exceeding the Limits of Visual-Linguistic Multi-Task Learning
• Composed Fine-Tuning: Freezing Pre-Trained Denoising Autoencoders for Improved Generalization
• Neural Network Surgery with Sets
• Improved Customer Transaction Classification using Semi-Supervised Knowledge Distillation
• Parameter-Efficient Methods for Metastases Detection from Clinical Notes
• The Efficiency Misnomer
• Domain Adaptor Networks for Hyperspectral Image Recognition
• Self-Tuning for Data-Efficient Deep Learning
• Analyzing Commonsense Emergence in Few-shot Knowledge Models
• Cheap Learning: Maximising Performance of Language Models for Social Data Science Using Minimal Data
• CERM: Context-aware Literature-based Discovery via Sentiment Analysis
• Neural Architecture Search for Sentence Classification with BERT