Hard Gate Knowledge Distillation -- Leverage Calibration for Robust and Reliable Language Model

Dongkyu Lee, Zhiliang Tian, Yingxiu Zhao +2

In knowledge distillation, a student model is trained with supervisions from both knowledge from a teacher and observations drawn from a training data distribution. Knowledge of a…