Robust Knowledge Distillation from RNN-T Models With Noisy Training Labels Using Full-Sum Loss

Mohammad Zeineldeen, Kartik Audhkhasi, Murali Karthick Baskar +1

This work studies knowledge distillation (KD) and addresses its constraints for recurrent neural network transducer (RNN-T) models. In hard distillation, a teacher model transcribe…