Gap Preserving Distillation by Building Bidirectional Mappings with A Dynamic Teacher

Yong Guo, Shulian Zhang, Haolin Pan +3

Knowledge distillation aims to transfer knowledge from a large teacher model to a compact student counterpart, often coming with a significant performance gap between them. We find…