Speed of Light Exact Greedy Decoding for RNN-T Speech Recognition Models on GPU

Daniel Galvez, Vladimir Bataev, Hainan Xu +1

The vast majority of inference time for RNN Transducer (RNN-T) models today is spent on decoding. Current state-of-the-art RNN-T decoding implementations leave the GPU idle ~80% of…