LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compression

Abstract

BERT is a cutting-edge language representation model pre-trained by a large corpus, which achieves superior performances on various natural language understanding tasks. However, a major blocking issue of applying BERT to online services is that it is memory-intensive and leads to unsatisfactory latency of user requests, raising the necessity of model compression. Existing solutions leverage the knowledge distillation framework to learn a smaller model that imitates the behaviors of BERT. However, the training procedure of knowledge distillation is expensive itself as it requires sufficient training data to imitate the teacher model. In this paper, we address this issue by proposing a hybrid solution named LadaBERT (Lightweight adaptation of BERT through hybrid model compression), which combines the advantages of different model compression methods, including weight pruning, matrix factorization and knowledge distillation. LadaBERT achieves state-of-the-art accuracy on various public datasets while the training overheads can be reduced by an order of magnitude.

arXiv commentCOLING2020

References in corpus (11)

• Distilling the Knowledge in a Neural Network
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Binarized Neural Networks
• Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
• Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
• Model compression via distillation and quantization
• Large scale distributed neural network training through online distillation
• Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers
• Towards the Limit of Network Quantization
• On the Effectiveness of Low-Rank Matrix Factorization for LSTM Model Compression

Cited by in corpus (11)

• Applications and Techniques for Fast Machine Learning in Science
• I-BERT: Integer-only BERT Quantization
• KDLSQ-BERT: A Quantized Bert Combining Knowledge Distillation with Learned Step Size Quantization
• Utility is in the Eye of the User: A Critique of NLP Leaderboards
• TernaryBERT: Distillation-aware Ultra-low Bit BERT
• KroneckerBERT: Learning Kronecker Decomposition for Pre-trained Language Models via Knowledge Distillation
• Direction is what you need: Improving Word Embedding Compression in Large Language Models
• ROSITA: Refined BERT cOmpreSsion with InTegrAted techniques
• Classification-based Quality Estimation: Small and Efficient Models for Real-world Applications
• Weight Squeezing: Reparameterization for Knowledge Transfer and Model Compression
• BERMo: What can BERT learn from ELMo?