Adapting by Pruning: A Case Study on BERT
arXiv:2105.03343
Abstract
Adapting pre-trained neural models to downstream tasks has become the standard practice for obtaining high-quality models. In this work, we propose a novel model adaptation paradigm, adapting by pruning, which prunes neural connections in the pre-trained model to optimise the performance on the target task; all remaining connections have their weights intact. We formulate adapting-by-pruning as an optimisation problem with a differentiable loss and propose an efficient algorithm to prune the model. We prove that the algorithm is near-optimal under standard assumptions and apply the algorithm to adapt BERT to some GLUE tasks. Results suggest that our method can prune up to 50% weights in BERT while yielding similar performance compared to the fine-tuned full model. We also compare our method with other state-of-the-art pruning methods and study the topological differences of their obtained sub-networks.
References in corpus (8)
- To prune, or not to prune: exploring the efficacy of pruning for model compression
- Learning multiple visual domains with residual adapters
- The State of Sparsity in Deep Neural Networks
- Revisiting Few-sample BERT Fine-tuning
- Training with Quantization Noise for Extreme Model Compression
- What is the State of Neural Network Pruning?
- Winning the Lottery with Continuous Sparsification
- Disentangling Neural Architectures and Weights: A Case Study in Supervised Classification