EfficientBERT: Progressively Searching Multilayer Perceptron via Warm-up Knowledge Distillation

Chenhe Dong, Guangrun Wang, Hang Xu +3

Pre-trained language models have shown remarkable results on various NLP tasks. Nevertheless, due to their bulky size and slow inference speed, it is hard to deploy them on edge de…