Knowledge Distillation from Internal Representations
arXiv:1910.03723
Abstract
Knowledge distillation is typically conducted by training a small model (the student) to mimic a large and cumbersome model (the teacher). The idea is to compress the knowledge from the teacher by using its output probabilities as soft-labels to optimize the student. However, when the teacher is considerably large, there is no guarantee that the internal knowledge of the teacher will be transferred into the student; even if the student closely matches the soft-labels, its internal representations may be considerably different. This internal mismatch can undermine the generalization capabilities originally intended to be transferred from the teacher to the student. In this paper, we propose to distill the internal representations of a large model such as BERT into a simplified version of it. We formulate two ways to distill such representations and various algorithms to conduct the distillation. We experiment with datasets from the GLUE benchmark and consistently show that adding knowledge distillation from internal representations is a more powerful method than only using soft-label distillation.
To appear in AAAI-2020
References in corpus (6)
- Distilling the Knowledge in a Neural Network
- FitNets: Hints for Thin Deep Nets
- Cross-lingual Language Model Pretraining
- Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
- Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding
- Effective Quantization Methods for Recurrent Neural Networks