MKD: a Multi-Task Knowledge Distillation Approach for Pretrained Language Models

Abstract

Pretrained language models have led to significant performance gains in many NLP tasks. However, the intensive computing resources to train such models remain an issue. Knowledge distillation alleviates this problem by learning a light-weight student model. So far the distillation approaches are all task-specific. In this paper, we explore knowledge distillation under the multi-task learning setting. The student is jointly distilled across different tasks. It acquires more general representation capacity through multi-tasking distillation and can be further fine-tuned to improve the model in the target domain. Unlike other BERT distillation methods which specifically designed for Transformer-based architectures, we provide a general learning framework. Our approach is model agnostic and can be easily applied on different future teacher model architectures. We evaluate our approach on a Transformer-based and LSTM based student model. Compared to a strong, similarly LSTM-based approach, we achieve better quality under the same computational constraints. Compared to the present state of the art, we reach comparable results with much faster inference speed.

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• Efficient Estimation of Word Representations in Vector Space
• Distilling the Knowledge in a Neural Network
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Bidirectional Attention Flow for Machine Comprehension
• Do Deep Nets Really Need to be Deep?
• Dynamic Coattention Networks For Question Answering
• Semi-supervised Sequence Learning
• Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
• Multi-Task Deep Neural Networks for Natural Language Understanding
• Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding
• The Benefit of Multitask Representation Learning

Cited by in corpus (8)

• Pretrained Transformers for Text Ranking: BERT and Beyond
• Mokey: Enabling Narrow Fixed-Point Inference for Out-of-the-Box Floating-Point Transformer Models
• MixKD: Towards Efficient Distillation of Large-scale Language Models
• Compressed Deep Networks: Goodbye SVD, Hello Robust Low-Rank Approximation
• Meta-KD: A Meta Knowledge Distillation Framework for Language Model Compression across Domains
• MergeDistill: Merging Pre-trained Language Models using Distillation
• HRKD: Hierarchical Relational Knowledge Distillation for Cross-domain Language Model Compression
• Deep Learning Meets Projective Clustering