LRC-BERT: Latent-representation Contrastive Knowledge Distillation for Natural Language Understanding
arXiv:2012.07335
Abstract
The pre-training models such as BERT have achieved great results in various natural language processing problems. However, a large number of parameters need significant amounts of memory and the consumption of inference time, which makes it difficult to deploy them on edge devices. In this work, we propose a knowledge distillation method LRC-BERT based on contrastive learning to fit the output of the intermediate layer from the angular distance aspect, which is not considered by the existing distillation methods. Furthermore, we introduce a gradient perturbation-based training architecture in the training phase to increase the robustness of LRC-BERT, which is the first attempt in knowledge distillation. Additionally, in order to better capture the distribution characteristics of the intermediate layer, we design a two-stage training method for the total distillation loss. Finally, by verifying 8 datasets on the General Language Understanding Evaluation (GLUE) benchmark, the performance of the proposed LRC-BERT exceeds the existing state-of-the-art methods, which proves the effectiveness of our method.
References in corpus (12)
- Distilling the Knowledge in a Neural Network
- A Simple Framework for Contrastive Learning of Visual Representations
- XLNet: Generalized Autoregressive Pretraining for Language Understanding
- Pre-trained Models for Natural Language Processing: A Survey
- Compressing Deep Convolutional Networks using Vector Quantization
- SQuAD: 100,000+ Questions for Machine Comprehension of Text
- MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
- GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
- Well-Read Students Learn Better: On the Importance of Pre-training Compact Models
- Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
- MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
- Small and Practical BERT Models for Sequence Labeling