AD-KD: Attribution-Driven Knowledge Distillation for Language Model Compression

Siyue Wu, Hongzhan Chen, Xiaojun Quan +2

Knowledge distillation has attracted a great deal of interest recently to compress pre-trained language models. However, existing knowledge distillation methods suffer from two lim…