Do we need Label Regularization to Fine-tune Pre-trained Language Models?

Ivan Kobyzev, Aref Jafari, Mehdi Rezagholizadeh +5

Knowledge Distillation (KD) is a prominent neural model compression technique that heavily relies on teacher network predictions to guide the training of a student model. Consideri…