Multi-microphone Complex Spectral Mapping for Utterance-wise and Continuous Speech Separation
arXiv:2010.01703
Abstract
We propose multi-microphone complex spectral mapping, a simple way of applying deep learning for time-varying non-linear beamforming, for speaker separation in reverberant conditions. We aim at both speaker separation and dereverberation. Our study first investigates offline utterance-wise speaker separation and then extends to block-online continuous speech separation (CSS). Assuming a fixed array geometry between training and testing, we train deep neural networks (DNN) to predict the real and imaginary (RI) components of target speech at a reference microphone from the RI components of multiple microphones. We then integrate multi-microphone complex spectral mapping with minimum variance distortionless response (MVDR) beamforming and post-filtering to further improve separation, and combine it with frame-level speaker counting for block-online CSS. Although our system is trained on simulated room impulse responses (RIR) based on a fixed number of microphones arranged in a given geometry, it generalizes well to a real array with the same geometry. State-of-the-art separation performance is obtained on the simulated two-talker SMS-WSJ corpus and the real-recorded LibriCSS dataset.
arXiv comment14 pages, 6 figures. To appear in IEEE/ACM Transactions on Audio, Speech, and Language Processing. Sound demo https://zqwang7.github.io/demos/SMSWSJ_demo/taslp20_SMSWSJ_demo.html
References in corpus (10)
- An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
- Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation
- Sound Event Localization and Detection of Overlapping Sources Using Convolutional Recurrent Neural Networks
- Multi-Speaker DOA Estimation Using Deep Convolutional Networks Trained with Noise Signals
- SMS-WSJ: Database, performance measures, and baseline recipe for multi-channel source separation and recognition
- On End-to-end Multi-channel Time Domain Speech Separation in Reverberant Environments
- Multi-talker ASR for an unknown number of sources: Joint training of source counting, separation and ASR
- Semantic Mask for Transformer based End-to-End Speech Recognition
- Continuous Speech Separation with Conformer
- ADL-MVDR: All deep learning MVDR beamformer for target speech separation