End-to-end training of time domain audio separation and recognition

Abstract

The rising interest in single-channel multi-speaker speech separation sparked development of End-to-End (E2E) approaches to multi-speaker speech recognition. However, up until now, state-of-the-art neural network-based time domain source separation has not yet been combined with E2E speech recognition. We here demonstrate how to combine a separation module based on a Convolutional Time domain Audio Separation Network (Conv-TasNet) with an E2E speech recognizer and how to train such a model jointly by distributing it over multiple GPUs or by approximating truncated back-propagation for the convolutional front-end. To put this work into perspective and illustrate the complexity of the design space, we provide a compact overview of single-channel multi-speaker recognition systems. Our experiments show a word error rate of 11.0% on WSJ0-2mix and indicate that our joint time domain model can yield substantial improvements over cascade DNN-HMM and monolithic E2E frequency domain systems proposed so far.

arXiv comment5 pages, 1 figure, to appear in ICASSP 2020

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• ADADELTA: An Adaptive Learning Rate Method
• Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation
• The Microsoft 2017 Conversational Speech Recognition System
• ESPnet: End-to-End Speech Processing Toolkit
• Analysis of Deep Clustering as Preprocessing for Automatic Speech Recognition of Sparsely Overlapping Speech
• Recognizing Multi-talker Speech with Permutation Invariant Training

Cited by in corpus (12)

• A Multi-Phase Gammatone Filterbank for Speech Separation via TasNet
• Multi-talker ASR for an unknown number of sources: Joint training of source counting, separation and ASR
• End-to-End Integration of Speech Separation and Voice Activity Detection for Low-Latency Diarization of Telephone Conversations
• Sandglasset: A Light Multi-Granularity Self-attentive Network For Time-Domain Speech Separation
• Streaming Multi-speaker ASR with RNN-T
• Multi-Speaker ASR Combining Non-Autoregressive Conformer CTC and Conditional Speaker Chain
• Multi-path RNN for hierarchical modeling of long sequential data and its application to speaker stream separation
• Audio-visual Multi-channel Integration and Recognition of Overlapped Speech
• Unifying Diarization, Separation, and ASR with Multi-Speaker Encoder
• Audio-visual Multi-channel Recognition of Overlapped Speech
• Investigation of Practical Aspects of Single Channel Speech Separation for ASR
• End-to-end Music-mixed Speech Recognition