Reweighted Proximal Pruning for Large-Scale Language Representation

Abstract

Recently, pre-trained language representation flourishes as the mainstay of the natural language understanding community, e.g., BERT. These pre-trained language representations can create state-of-the-art results on a wide range of downstream tasks. Along with continuous significant performance improvement, the size and complexity of these pre-trained neural models continue to increase rapidly. Is it possible to compress these large-scale language representation models? How will the pruned language representation affect the downstream multi-task transfer learning objectives? In this paper, we propose Reweighted Proximal Pruning (RPP), a new pruning method specifically designed for a large-scale language representation model. Through experiments on SQuAD and the GLUE benchmark suite, we show that proximal pruned BERT keeps high accuracy for both the pre-training task and the downstream multiple fine-tuning tasks at high prune ratio. RPP provides a new perspective to help us analyze what large-scale language representation might learn. Additionally, RPP makes it possible to deploy a large state-of-the-art language representation model such as BERT on a series of distinct devices (e.g., online servers, mobile phones, and edge devices).

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Decoupled Weight Decay Regularization
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• Rethinking the Value of Network Pruning
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• Pruning Filters for Efficient ConvNets
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• Semi-supervised Sequence Learning
• A large annotated corpus for learning natural language inference
• Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT
• Are Sixteen Heads Really Better than One?
• Multi-Granularity Self-Attention for Neural Machine Translation

Cited by in corpus (16)

• Chasing Sparsity in Vision Transformers: An End-to-End Exploration
• Only Train Once: A One-Shot Neural Network Training And Pruning Framework
• The Lottery Ticket Hypothesis for Pre-trained BERT Networks
• Movement Pruning: Adaptive Sparsity by Fine-Tuning
• Real-Time Execution of Large-scale Language Models on Mobile
• AUBER: Automated BERT Regularization
• NxMTransformer: Semi-Structured Sparsification for Natural Language Understanding via ADMM
• Rethinking Network Pruning -- under the Pre-train and Fine-tune Paradigm
• Which *BERT? A Survey Organizing Contextualized Encoders
• Contrastive Distillation on Intermediate Representations for Language Model Compression
• Compressed Deep Networks: Goodbye SVD, Hello Robust Low-Rank Approximation
• Structured Sparsity Inducing Adaptive Optimizers for Deep Learning
• Adversarial Self-Supervised Data-Free Distillation for Text Classification
• Deep Learning Meets Projective Clustering
• Weight Squeezing: Reparameterization for Knowledge Transfer and Model Compression
• Structured Pattern Pruning Using Regularization