Parameter-Efficient Transfer Learning with Diff Pruning

Abstract

While task-specific finetuning of pretrained networks has led to significant empirical advances in NLP, the large size of networks makes finetuning difficult to deploy in multi-task, memory-constrained settings. We propose diff pruning as a simple approach to enable parameter-efficient transfer learning within the pretrain-finetune framework. This approach views finetuning as learning a task-specific diff vector that is applied on top of the pretrained parameter vector, which remains fixed and is shared across different tasks. The diff vector is adaptively pruned during training with a differentiable approximation to the L0-norm penalty to encourage sparsity. Diff pruning becomes parameter-efficient as the number of tasks increases, as it requires storing only the nonzero positions and weights of the diff vector for each task, while the cost of storing the shared pretrained model remains constant. It further does not require access to all tasks during training, which makes it attractive in settings where tasks arrive in stream or the set of tasks is unknown. We find that models finetuned with diff pruning can match the performance of fully finetuned baselines on the GLUE benchmark while only modifying 0.5% of the pretrained model's parameters per task.

arXiv commentACL 2021

References in corpus (16)

• Efficient Estimation of Word Representations in Vector Space
• Overcoming catastrophic forgetting in neural networks
• Distributed Representations of Sentences and Documents
• Categorical Reparameterization with Gumbel-Softmax
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• Skip-Thought Vectors
• Semi-supervised Sequence Learning
• Well-Read Students Learn Better: On the Importance of Pre-training Compact Models
• Learning How to Ask: Querying LMs with Mixtures of Soft Prompts
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• Overcoming Catastrophic Forgetting by Incremental Moment Matching
• AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning
• BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning
• ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
• Movement Pruning: Adaptive Sparsity by Fine-Tuning

Cited by in corpus (7)

• Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey
• HiddenCut: Simple Data Augmentation for Natural Language Understanding with Better Generalization
• Training Neural Networks with Fixed Sparse Masks
• DSEE: Dually Sparsity-embedded Efficient Tuning of Pre-trained Language Models
• Efficient Test Time Adapter Ensembling for Low-resource Language Varieties
• A Survey on Green Deep Learning
• The Efficiency Misnomer