Acoustic Space Learning for Sound Source Separation and Localization on Binaural Manifolds

Abstract

In this paper we address the problems of modeling the acoustic space generated by a full-spectrum sound source and of using the learned model for the localization and separation of multiple sources that simultaneously emit sparse-spectrum sounds. We lay theoretical and methodological grounds in order to introduce the binaural manifold paradigm. We perform an in-depth study of the latent low-dimensional structure of the high-dimensional interaural spectral data, based on a corpus recorded with a human-like audiomotor robot head. A non-linear dimensionality reduction technique is used to show that these data lie on a two-dimensional (2D) smooth manifold parameterized by the motor states of the listener, or equivalently, the sound source directions. We propose a probabilistic piecewise affine mapping model (PPAM) specifically designed to deal with high-dimensional data exhibiting an intrinsic piecewise linear structure. We derive a closed-form expectation-maximization (EM) procedure for estimating the model parameters, followed by Bayes inversion for obtaining the full posterior density function of a sound source direction. We extend this solution to deal with missing data and redundancy in real world spectrograms, and hence for 2D localization of natural sound sources such as speech. We further generalize the model to the challenging case of multiple sound sources and we propose a variational EM framework. The associated algorithm, referred to as variational EM for source separation and localization (VESSL) yields a Bayesian estimation of the 2D locations and time-frequency masks of all the sources. Comparisons of the proposed approach with several existing methods reveal that the combination of acoustic-space learning with Bayesian inference enables our method to outperform state-of-the-art methods.

19 pages, 9 figures, 3 tables

Cited by in corpus (11)

• Estimation of the Direct-Path Relative Transfer Function for Supervised Sound-Source Localization
• Co-Localization of Audio Sources in Images Using Binaural Features and Locally-Linear Regression
• Audio-Based Search and Rescue with a Drone: Highlights from the IEEE Signal Processing Cup 2019 Student Competition
• Multiple-Speaker Localization Based on Direct-Path Features and Likelihood Maximization with Spatial Sparsity Regularization
• Reverberant Sound Localization with a Robot Head Based on Direct-Path Relative Transfer Function
• BeamLearning: an end-to-end Deep Learning approach for the angular localization of sound sources using raw multichannel acoustic pressure data
• Approximations of conditional probability density functions in Lebesgue spaces via mixture of experts models
• Mean absorption estimation from room impulse responses using virtually supervised learning
• A Review on Sound Source Localization in Robotics: Focusing on Deep Learning Methods
• Modeling the Comb Filter Effect and Interaural Coherence for Binaural Source Separation
• Rectified binaural ratio: A complex T-distributed feature for robust sound localization