Distribution Shift Matters for Knowledge Distillation with Webly Collected Images

Jialiang Tang, Shuo Chen, Gang Niu +2

Knowledge distillation aims to learn a lightweight student network from a pre-trained teacher network. In practice, existing knowledge distillation methods are usually infeasible w…