Weight-Inherited Distillation for Task-Agnostic BERT Compression

Taiqiang Wu, Cheng Hou, Shanshan Lao +4

Knowledge Distillation (KD) is a predominant approach for BERT compression. Previous KD-based methods focus on designing extra alignment losses for the student model to mimic the b…