Multi-Level Decoupled Relational Distillation for Heterogeneous Architectures

Yaoxin Yang, Peng Ye, Weihao Lin +4

Heterogeneous distillation is an effective way to transfer knowledge from cross-architecture teacher models to student models. However, existing heterogeneous distillation methods…