f-Divergence Minimization for Sequence-Level Knowledge Distillation

Yuqiao Wen, Zichao Li, Wenyu Du +1

Knowledge distillation (KD) is the process of transferring knowledge from a large model to a small one. It has gained increasing attention in the natural language processing commun…