Function-Consistent Feature Distillation

Dongyang Liu, Meina Kan, Shiguang Shan +1

Feature distillation makes the student mimic the intermediate features of the teacher. Nearly all existing feature-distillation methods use L2 distance or its slight variants as th…