Aligning in a Compact Space: Contrastive Knowledge Distillation between Heterogeneous Architectures

Hongjun Wu, Li Xiao, Xingkuo Zhang +1

Knowledge distillation is commonly employed to compress neural networks, reducing the inference costs and memory footprint. In the scenario of homogenous architecture, feature-base…