Improving RNN Transducer With Target Speaker Extraction and Neural Uncertainty Estimation

Abstract

Target-speaker speech recognition aims to recognize target-speaker speech from noisy environments with background noise and interfering speakers. This work presents a joint framework that combines time-domain target-speaker speech extraction and Recurrent Neural Network Transducer (RNN-T). To stabilize the joint-training, we propose a multi-stage training strategy that pre-trains and fine-tunes each module in the system before joint-training. Meanwhile, speaker identity and speech enhancement uncertainty measures are proposed to compensate for residual noise and artifacts from the target speech extraction module. Compared to a recognizer fine-tuned with a target speech extraction model, our experiments show that adding the neural uncertainty module significantly reduces 17% relative Character Error Rate (CER) on multi-speaker signals with background noise. The multi-condition experiments indicate that our method can achieve 9% relative performance gain in the noisy condition while maintaining the performance in the clean condition.

arXiv commentAccepted by ICASSP2021

References in corpus (7)

• Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation
• Sequence Transduction with Recurrent Neural Networks
• A Comparative Study on Transformer vs RNN in Speech Applications
• AISHELL-2: Transforming Mandarin ASR Research Into Industrial Scale
• Multichannel End-to-end Speech Recognition
• Multi-talker ASR for an unknown number of sources: Joint training of source counting, separation and ASR
• VoiceFilter-Lite: Streaming Targeted Voice Separation for On-Device Speech Recognition