Search to Distill: Pearls are Everywhere but not the Eyes

Yu Liu, Xuhui Jia, Mingxing Tan +4

Standard Knowledge Distillation (KD) approaches distill the knowledge of a cumbersome teacher model into the parameters of a student model with a pre-defined architecture. However,…