Voice Separation with an Unknown Number of Multiple Speakers

Abstract

We present a new method for separating a mixed audio sequence, in which multiple voices speak simultaneously. The new method employs gated neural networks that are trained to separate the voices at multiple processing steps, while maintaining the speaker in each output channel fixed. A different model is trained for every number of possible speakers, and the model with the largest number of speakers is employed to select the actual number of speakers in a given sample. Our method greatly outperforms the current state of the art, which, as we show, is not competitive for more than two speakers.

Accepted to ICML 2020. For associated audio samples, see http://enk100.github.io/speaker_separation

References in corpus (5)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation
• Deep attractor network for single-microphone speaker separation
• Speaker-independent Speech Separation with Deep Attractor Network
• Music Source Separation in the Waveform Domain

Cited by in corpus (20)

• Music Source Separation in the Waveform Domain
• Unsupervised Sound Separation Using Mixture Invariant Training
• D3Net: Densely connected multidilated DenseNet for music source separation
• Multi-talker ASR for an unknown number of sources: Joint training of source counting, separation and ASR
• The Cone of Silence: Speech Separation by Localization
• Separate but Together: Unsupervised Federated Learning for Speech Enhancement from Non-IID Data
• Towards Listening to 10 People Simultaneously: An Efficient Permutation Invariant Training of Audio Source Separation Using Sinkhorn's Algorithm
• Stepwise-Refining Speech Separation Network via Fine-Grained Encoding in High-order Latent Domain
• Attention is All You Need in Speech Separation
• Time-Domain Speech Extraction with Spatial Information and Multi Speaker Conditioning Mechanism
• Exploring the time-domain deep attractor network with two-stream architectures in a reverberant environment
• LaFurca: Iterative Refined Speech Separation Based on Context-Aware Dual-Path Parallel Bi-LSTM
• Improved Source Counting and Separation for Monaural Mixture
• Densely connected multidilated convolutional networks for dense prediction tasks
• One Shot Learning for Speech Separation
• Unified Gradient Reweighting for Model Biasing with Applications to Source Separation
• Independent Vector Analysis with Deep Neural Network Source Priors
• Semi-Supervised Singing Voice Separation with Noisy Self-Training
• Manifold-Aware Deep Clustering: Maximizing Angles between Embedding Vectors Based on Regular Simplex
• Toward Speech Separation in The Pre-Cocktail Party Problem with TasTas