Bridging the Gap between Decision and Logits in Decision-based Knowledge Distillation for Pre-trained Language Models

Qinhong Zhou, Zonghan Yang, Peng Li +1

Conventional knowledge distillation (KD) methods require access to the internal information of teachers, e.g., logits. However, such information may not always be accessible for la…