Graph-based Knowledge Distillation by Multi-head Attention Network

Seunghyun Lee, Byung Cheol Song

Knowledge distillation (KD) is a technique to derive optimal performance from a small student network (SN) by distilling knowledge of a large teacher network (TN) and transferring…