Fuse Before Transfer: Knowledge Fusion for Heterogeneous Distillation

Guopeng Li, Qiang Wang, Ke Yan +3

Most knowledge distillation (KD) methodologies predominantly focus on teacher-student pairs with similar architectures, such as both being convolutional neural networks (CNNs). How…