Improving Knowledge Distillation for BERT Models: Loss Functions, Mapping Methods, and Weight Tuning

Apoorv Dankar, Adeem Jassani, Kartikaeya Kumar

The use of large transformer-based models such as BERT, GPT, and T5 has led to significant advancements in natural language processing. However, these models are computationally ex…