A Survey of Sound Source Localization with Deep Learning Methods

Abstract

This article is a survey on deep learning methods for single and multiple sound source localization. We are particularly interested in sound source localization in indoor/domestic environment, where reverberation and diffuse noise are present. We provide an exhaustive topography of the neural-based localization literature in this context, organized according to several aspects: the neural network architecture, the type of input features, the output strategy (classification or regression), the types of data used for model training and evaluation, and the model training strategy. This way, an interested reader can easily comprehend the vast panorama of the deep learning-based sound source localization methods. Tables summarizing the literature survey are provided at the end of the paper for a quick search of methods with a given set of target characteristics.

arXiv commentAccepted for publication in The Journal of the Acoustical Society of America

References in corpus (75)

• Auto-Encoding Variational Bayes
• Neural Machine Translation by Jointly Learning to Align and Translate
• mixup: Beyond Empirical Risk Minimization
• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• An Overview of Multi-Task Learning in Deep Neural Networks
• Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation
• Deep Convolutional Neural Networks and Data Augmentation for Environmental Sound Classification
• Multiple Object Tracking: A Literature Review
• Deep Learning for Audio Signal Processing
• Deep Learning in Video Multi-Object Tracking: A Survey
• Machine learning in acoustics: theory and applications
• Pyroomacoustics: A Python package for audio room simulations and array processing algorithms
• Sound Event Localization and Detection of Overlapping Sources Using Convolutional Recurrent Neural Networks
• Conformer: Convolution-augmented Transformer for Speech Recognition
• The Cosparse Analysis Model and Algorithms
• Multi Snapshot Sparse Bayesian Learning for DOA Estimation
• Multi-Speaker DOA Estimation Using Deep Convolutional Networks Trained with Noise Signals
• Broadband DOA estimation using Convolutional neural networks trained with noise signals
• DeepMUSIC: Multiple Signal Classification via Deep Learning
• Deep Neural Networks for Multiple Speaker Detection and Localization
• Gauge Equivariant Convolutional Networks and the Icosahedral CNN
• Dynamical Variational Autoencoders: A Comprehensive Review
• Grid-free compressive beamforming
• The LOCATA Challenge: Acoustic Source Localization and Tracking
• Overview and Evaluation of Sound Event Localization and Detection in DCASE 2019
• gpuRIR: A Python Library for Room Impulse Response Simulation with GPU Acceleration
• Building and Evaluation of a Real Room Impulse Response Dataset
• Polyphonic Sound Event Detection and Localization using a Two-Stage Strategy
• Towards End-to-End Acoustic Localization using Deep Learning: from Audio Signal to Source Position Coordinates
• Robust Sound Source Tracking Using SRP-PHAT and 3D Convolutional Neural Networks
• Sample-level Deep Convolutional Neural Networks for Music Auto-tagging Using Raw Waveforms
• Deep Long Short-Term Memory Adaptive Beamforming Networks For Multichannel Robust Speech Recognition
• Trellis Networks for Sequence Modeling
• Semi-supervised multichannel speech enhancement with variational autoencoders and non-negative matrix factorization
• Quaternion Convolutional Neural Networks for Detection and Localization of 3D Sound Events
• SELD-TCN: Sound Event Localization & Detection via Temporal Convolutional Networks
• Estimation of the Direct-Path Relative Transfer Function for Supervised Sound-Source Localization
• Co-Localization of Audio Sources in Images Using Binaural Features and Locally-Linear Regression
• Regression and Classification for Direction-of-Arrival Estimation with Convolutional Recurrent Neural Networks
• A Sequence Matching Network for Polyphonic Sound Event Localization and Detection
• A Dataset of Reverberant Spatial Sound Scenes with Moving Sources for Sound Event Localization and Detection
• Multiple-Speaker Localization Based on Direct-Path Features and Likelihood Maximization with Spatial Sparsity Regularization
• Multi-Speaker Localization Using Convolutional Neural Network Trained with Noise
• Semi-supervised source localization with deep generative modeling
• Reverberant Sound Localization with a Robot Head Based on Direct-Path Relative Transfer Function
• The Cone of Silence: Speech Separation by Localization
• A Four-Stage Data Augmentation Approach to ResNet-Conformer Based Acoustic Modeling for Sound Event Localization and Detection
• Multi-Source DOA Estimation through Pattern Recognition of the Modal Coherence of a Reverberant Soundfield
• Event-Independent Network for Polyphonic Sound Event Localization and Detection
• BeamLearning: an end-to-end Deep Learning approach for the angular localization of sound sources using raw multichannel acoustic pressure data
• A Dataset of Dynamic Reverberant Sound Scenes with Directional Interferers for Sound Event Localization and Detection
• Ensemble of ACCDOA- and EINV2-based Systems with D3Nets and Impulse Response Simulation for Sound Event Localization and Detection
• On Multitask Loss Function for Audio Event Detection and Localization
• TRAMP: Tracking by a Real-time AMbisonic-based Particle filter
• Deep Residual Network for Sound Source Localization in the Time Domain
• ACCDOA: Activity-Coupled Cartesian Direction of Arrival Representation for Sound Event Localization and Detection
• Learning and controlling the source-filter representation of speech with a variational autoencoder
• Sound Event Localization and Detection Using Activity-Coupled Cartesian DOA Vector and RD3net
• Deep Sound Field Reconstruction in Real Rooms: Introducing the ISOBEL Sound Field Dataset
• SoundDet: Polyphonic Moving Sound Event Detection and Localization from Raw Waveform
• An Improved Event-Independent Network for Polyphonic Sound Event Localization and Detection
• End-to-end Microphone Permutation and Number Invariant Multi-channel Speech Separation
• MMDenseLSTM: An efficient combination of convolutional and recurrent neural networks for audio source separation
• Multi-scale aggregation of phase information for reducing computational cost of CNN based DOA estimation
• A General Network Architecture for Sound Event Localization and Detection Using Transfer Learning and Recurrent Neural Network
• SSLIDE: Sound Source Localization for Indoors based on Deep Learning
• Tracking by Animation: Unsupervised Learning of Multi-Object Attentive Trackers
• First Order Ambisonics Domain Spatial Augmentation for DNN-based Direction of Arrival Estimation
• Directional ASR: A New Paradigm for E2E Multi-Speaker Speech Recognition with Source Localization
• Data-Efficient Framework for Real-world Multiple Sound Source 2D Localization
• Sound event localization and detection based on crnn using rectangular filters and channel rotation data augmentation
• Efficient Training Data Generation for Phase-Based DOA Estimation
• Sound Event Localization and Detection using Squeeze-Excitation Residual CNNs
• Self-supervised Neural Audio-Visual Sound Source Localization via Probabilistic Spatial Modeling

Cited by in corpus (21)

• DA-MUSIC: Data-Driven DoA Estimation via Deep Augmented MUSIC Algorithm
• Direction of Arrival Estimation of Sound Sources Using Icosahedral CNNs
• GWA: A Large High-Quality Acoustic Dataset for Audio Processing
• Extending GCC-PHAT using Shift Equivariant Neural Networks
• Machine Learning in Acoustics: A Review and Open-Source Repository
• A convolutional plane wave model for sound field reconstruction
• Enemy Spotted: in-game gun sound dataset for gunshot classification and localization
• A Review on Sound Source Localization in Robotics: Focusing on Deep Learning Methods
• Assisted RTF-Vector-Based Binaural Direction of Arrival Estimation Exploiting a Calibrated External Microphone Array
• BERP: A Blind Estimator of Room Parameters for Single-Channel Noisy Speech Signals
• wav2pos: Sound Source Localization using Masked Autoencoders
• Spatial LibriSpeech: An Augmented Dataset for Spatial Audio Learning
• Neural network for multi-exponential sound energy decay analysis
• Scalable-Complexity Steered Response Power Mapping based on Low-Rank and Sparse Interpolation
• Audiovisual Saliency Prediction in Uncategorized Video Sequences based on Audio-Video Correlation
• Exploiting an External Microphone for Binaural RTF-Vector-Based Direction of Arrival Estimation for Multiple Speakers
• CNN-based Robust Sound Source Localization with SRP-PHAT for the Extreme Edge
• How far generated data can impact Neural Networks performance?
• Study of speaker localization with binaural microphone array incorporating auditory filters and lateral angle estimation
• An experiment on an automated literature survey of data-driven speech enhancement methods
• Completing Sets of Prototype Transfer Functions for Subspace-based Direction of Arrival Estimation of Multiple Speakers