Knowledge Distillation of Transformer-based Language Models Revisited

Chengqiang Lu, Jianwei Zhang, Yunfei Chu +5

In the past few years, transformer-based pre-trained language models have achieved astounding success in both industry and academia. However, the large model size and high run-time…