On Elastic Language Models

Chen Zhang, Benyou Wang, Dawei Song

Large-scale pretrained language models have achieved compelling performance in a wide range of language understanding and information retrieval tasks. Knowledge distillation offers…