AdaKD: Dynamic Knowledge Distillation of ASR models using Adaptive Loss Weighting

Shreyan Ganguly, Roshan Nayak, Rakshith Rao +2

Knowledge distillation, a widely used model compression technique, works on the basis of transferring knowledge from a cumbersome teacher model to a lightweight student model. The…