Can Students Beyond The Teacher? Distilling Knowledge from Teacher's Bias

Jianhua Zhang, Yi Gao, Ruyu Liu +3

Knowledge distillation (KD) is a model compression technique that transfers knowledge from a large teacher model to a smaller student model to enhance its performance. Existing met…