The Cone of Silence: Speech Separation by Localization

Abstract

Given a multi-microphone recording of an unknown number of speakers talking concurrently, we simultaneously localize the sources and separate the individual speakers. At the core of our method is a deep network, in the waveform domain, which isolates sources within an angular region $\theta \pm w/2$, given an angle of interest $\theta$ and angular window size $w$. By exponentially decreasing $w$, we can perform a binary search to localize and separate all sources in logarithmic time. Our algorithm allows for an arbitrary number of potentially moving speakers at test time, including more speakers than seen during training. Experiments demonstrate state-of-the-art performance for both source separation and source localization, particularly in high levels of background noise.

arXiv comment9 pages + references + supplementary. Oral presentation at NeurIPS 2020

References in corpus (11)

• Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation
• Sound Event Localization and Detection of Overlapping Sources Using Convolutional Recurrent Neural Networks
• Supervised and Unsupervised Speech Enhancement Using Nonnegative Matrix Factorization
• Robust Sound Source Localization Using a Microphone Array on a Mobile Robot
• Deep Neural Networks for Multiple Speaker Detection and Localization
• Deep Clustering and Conventional Networks for Music Separation: Stronger Together
• Exploiting Deep Neural Networks and Head Movements for Robust Binaural Localisation of Multiple Sources in Reverberant Environments
• Voice Separation with an Unknown Number of Multiple Speakers
• Demucs: Deep Extractor for Music Sources with extra unlabeled data remixed
• Unsupervised Deep Clustering for Source Separation: Direct Learning from Mixtures using Spatial Information
• Enhancing End-to-End Multi-channel Speech Separation via Spatial Feature Learning

Cited by in corpus (5)

• A Survey of Sound Source Localization with Deep Learning Methods
• ClearBuds: Wireless Binaural Earbuds for Learning-Based Speech Enhancement
• GWA: A Large High-Quality Acoustic Dataset for Audio Processing
• Sound Synthesis, Propagation, and Rendering: A Survey
• A Review on Sound Source Localization in Robotics: Focusing on Deep Learning Methods