Co-Localization of Audio Sources in Images Using Binaural Features and Locally-Linear Regression

Abstract

This paper addresses the problem of localizing audio sources using binaural measurements. We propose a supervised formulation that simultaneously localizes multiple sources at different locations. The approach is intrinsically efficient because, contrary to prior work, it relies neither on source separation, nor on monaural segregation. The method starts with a training stage that establishes a locally-linear Gaussian regression model between the directional coordinates of all the sources and the auditory features extracted from binaural measurements. While fixed-length wide-spectrum sounds (white noise) are used for training to reliably estimate the model parameters, we show that the testing (localization) can be extended to variable-length sparse-spectrum sounds (such as speech), thus enabling a wide range of realistic applications. Indeed, we demonstrate that the method can be used for audio-visual fusion, namely to map speech signals onto images and hence to spatially align the audio and visual modalities, thus enabling to discriminate between speaking and non-speaking faces. We release a novel corpus of real-room recordings that allow quantitative evaluation of the co-localization method in the presence of one or two sound sources. Experiments demonstrate increased accuracy and speed relative to several state-of-the-art methods.

15 pages, 8 figures

References in corpus (3)

• Fisher Lecture: Dimension Reduction in Regression
• Acoustic Space Learning for Sound Source Separation and Localization on Binaural Manifolds
• High-Dimensional Regression with Gaussian Mixtures and Partially-Latent Response Variables

Cited by in corpus (11)

• A Survey of Sound Source Localization with Deep Learning Methods
• EM Algorithms for Weighted-Data Clustering with Application to Audio-Visual Scene Analysis
• Audio-Visual Speaker Diarization Based on Spatiotemporal Bayesian Fusion
• Estimation of the Direct-Path Relative Transfer Function for Supervised Sound-Source Localization
• Multiple-Speaker Localization Based on Direct-Path Features and Likelihood Maximization with Spatial Sparsity Regularization
• Reverberant Sound Localization with a Robot Head Based on Direct-Path Relative Transfer Function
• Mean absorption estimation from room impulse responses using virtually supervised learning
• Variational Bayesian Inference for Audio-Visual Tracking of Multiple Speakers
• Multi-target DoA Estimation with an Audio-visual Fusion Mechanism
• Tragic Talkers: A Shakespearean Sound- and Light-Field Dataset for Audio-Visual Machine Learning Research
• Relative Transfer Function Inverse Regression from Low Dimensional Manifold