SKDBERT: Compressing BERT via Stochastic Knowledge Distillation

Zixiang Ding, Guoqing Jiang, Shuai Zhang +2

In this paper, we propose Stochastic Knowledge Distillation (SKD) to obtain compact BERT-style language model dubbed SKDBERT. In each iteration, SKD samples a teacher model from a…