MSD: Saliency-aware Knowledge Distillation for Multimodal Understanding

Woojeong Jin, Maziar Sanjabi, Shaoliang Nie +3

To reduce a model size but retain performance, we often rely on knowledge distillation (KD) which transfers knowledge from a large "teacher" model to a smaller "student" model. How…