Gradient Knowledge Distillation for Pre-trained Language Models

Lean Wang, Lei Li, Xu Sun

Knowledge distillation (KD) is an effective framework to transfer knowledge from a large-scale teacher to a compact yet well-performing student. Previous KD practices for pre-train…