Accelerating Large Scale Knowledge Distillation via Dynamic Importance Sampling

Minghan Li, Tanli Zuo, Ruicheng Li +2

Knowledge distillation is an effective technique that transfers knowledge from a large teacher model to a shallow student. However, just like massive classification, large scale kn…