Extracting knowledge from features with multilevel abstraction

Jinhong Lin, Zhaoyang Li

Knowledge distillation aims at transferring the knowledge from a large teacher model to a small student model with great improvements of the performance of the student model. There…