Anti-Shortcut Distillation via Temporal Negative Knowledge Transfer

Syed Muhammad Raza, Omer Tariq, Jeongbae Son

Knowledge distillation (KD) trains a compact student by attracting it towards a converged teacher. It is silent about which directions the teacher itself learned to suppress: repul…