Which Student is Best? A Comprehensive Knowledge Distillation Exam for Task-Specific BERT Models

Made Nindyatama Nityasya, Haryo Akbarianto Wibowo, Rendi Chevi +2

We perform knowledge distillation (KD) benchmark from task-specific BERT-base teacher models to various student models: BiLSTM, CNN, BERT-Tiny, BERT-Mini, and BERT-Small. Our exper…