MLKD-BERT: Multi-level Knowledge Distillation for Pre-trained Language Models

Ying Zhang, Ziheng Yang, Shufan Ji

Knowledge distillation is an effective technique for pre-trained language model compression. Although existing knowledge distillation methods perform well for the most typical mode…