Learning to Project for Cross-Task Knowledge Distillation

Dylan Auty, Roy Miles, Benedikt Kolbeinsson +1

Traditional knowledge distillation (KD) relies on a proficient teacher trained on the target task, which is not always available. In this setting, cross-task distillation can be us…