Multi-talker ASR for an unknown number of sources: Joint training of source counting, separation and ASR

Abstract

Most approaches to multi-talker overlapped speech separation and recognition assume that the number of simultaneously active speakers is given, but in realistic situations, it is typically unknown. To cope with this, we extend an iterative speech extraction system with mechanisms to count the number of sources and combine it with a single-talker speech recognizer to form the first end-to-end multi-talker automatic speech recognition system for an unknown number of active speakers. Our experiments show very promising performance in counting accuracy, source separation and speech recognition on simulated clean mixtures from WSJ0-2mix and WSJ0-3mix. Among others, we set a new state-of-the-art word error rate on the WSJ0-2mix database. Furthermore, our system generalizes well to a larger number of speakers than it ever saw during training, as shown in experiments with the WSJ0-4mix database.

arXiv comment5 pages, INTERSPEECH 2020

References in corpus (6)

• ADADELTA: An Adaptive Learning Rate Method
• Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation
• Voice Separation with an Unknown Number of Multiple Speakers
• End-to-end training of time domain audio separation and recognition
• Analysis of Deep Clustering as Preprocessing for Automatic Speech Recognition of Sparsely Overlapping Speech
• Serialized Output Training for End-to-End Overlapped Speech Recognition

Cited by in corpus (8)

• Sparsely Overlapped Speech Training in the Time Domain: Joint Learning of Target Speech Separation and Personal VAD Benefits
• ESPnet-se: end-to-end speech enhancement and separation toolkit designed for asr integration
• Unifying Diarization, Separation, and ASR with Multi-Speaker Encoder
• Investigation of Practical Aspects of Single Channel Speech Separation for ASR
• Dual-Path Modeling for Long Recording Speech Separation in Meetings
• Multi-microphone Complex Spectral Mapping for Utterance-wise and Continuous Speech Separation
• Exploring End-to-End Multi-channel ASR with Bias Information for Meeting Transcription
• Improving RNN Transducer With Target Speaker Extraction and Neural Uncertainty Estimation