Sound Event Localization and Detection of Overlapping Sources Using Convolutional Recurrent Neural Networks

Abstract

In this paper, we propose a convolutional recurrent neural network for joint sound event localization and detection (SELD) of multiple overlapping sound events in three-dimensional (3D) space. The proposed network takes a sequence of consecutive spectrogram time-frames as input and maps it to two outputs in parallel. As the first output, the sound event detection (SED) is performed as a multi-label classification task on each time-frame producing temporal activity for all the sound event classes. As the second output, localization is performed by estimating the 3D Cartesian coordinates of the direction-of-arrival (DOA) for each sound event class using multi-output regression. The proposed method is able to associate multiple DOAs with respective sound event labels and further track this association with respect to time. The proposed method uses separately the phase and magnitude component of the spectrogram calculated on each audio channel as the feature, thereby avoiding any method- and array-specific feature extraction. The method is evaluated on five Ambisonic and two circular array format datasets with different overlapping sound events in anechoic, reverberant and real-life scenarios. The proposed method is compared with two SED, three DOA estimation, and one SELD baselines. The results show that the proposed method is generic and applicable to any array structures, robust to unseen DOA values, reverberation, and low SNR scenarios. The proposed method achieved a consistently higher recall of the estimated number of DOAs across datasets in comparison to the best baseline. Additionally, this recall was observed to be significantly better than the best baseline method for a higher number of overlapping sound events.

arXiv commentPublished in Journal of Selected Topics in Signal Processing 2018

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Convolutional Recurrent Neural Networks for Polyphonic Sound Event Detection
• Recurrent Neural Networks for Polyphonic Sound Event Detection in Real Life Recordings
• Broadband DOA estimation using Convolutional neural networks trained with noise signals
• Robust Localization and Tracking of Simultaneous Moving Sound Sources Using Beamforming and Particle Filtering
• Deep Neural Networks for Multiple Speaker Detection and Localization
• Sound Event Detection in Multichannel Audio Using Spatial and Harmonic Features
• A report on sound event detection with different binaural features
• Multi-Speaker Localization Using Convolutional Neural Network Trained with Noise

Cited by in corpus (59)

• Machine learning in acoustics: theory and applications
• A Survey of Sound Source Localization with Deep Learning Methods
• Overview and Evaluation of Sound Event Localization and Detection in DCASE 2019
• Polyphonic Sound Event Detection and Localization using a Two-Stage Strategy
• Robust Sound Source Tracking Using SRP-PHAT and 3D Convolutional Neural Networks
• Sound source detection, localization and classification using consecutive ensemble of CRNN models
• Quaternion Convolutional Neural Networks for Detection and Localization of 3D Sound Events
• SALSA: Spatial Cue-Augmented Log-Spectrogram Features for Polyphonic Sound Event Localization and Detection
• Regression and Classification for Direction-of-Arrival Estimation with Convolutional Recurrent Neural Networks
• Upper Esophageal Sphincter Opening Segmentation with Convolutional Recurrent Neural Networks in High Resolution Cervical Auscultation
• A Sequence Matching Network for Polyphonic Sound Event Localization and Detection
• L3DAS22 Challenge: Learning 3D Audio Sources in a Real Office Environment
• A Dataset of Reverberant Spatial Sound Scenes with Moving Sources for Sound Event Localization and Detection
• SALSA-Lite: A Fast and Effective Feature for Polyphonic Sound Event Localization and Detection with Microphone Arrays
• PHNNs: Lightweight Neural Networks via Parameterized Hypercomplex Convolutions
• Direction of Arrival Estimation of Sound Sources Using Icosahedral CNNs
• Cross-task learning for audio tagging, sound event detection and spatial localization: DCASE 2019 baseline systems
• Semi-supervised source localization with deep generative modeling
• The Cone of Silence: Speech Separation by Localization
• Event-Independent Network for Polyphonic Sound Event Localization and Detection
• A Four-Stage Data Augmentation Approach to ResNet-Conformer Based Acoustic Modeling for Sound Event Localization and Detection
• Multi-Source DOA Estimation through Pattern Recognition of the Modal Coherence of a Reverberant Soundfield
• Speaker Distance Estimation in Enclosures from Single-Channel Audio
• BeamLearning: an end-to-end Deep Learning approach for the angular localization of sound sources using raw multichannel acoustic pressure data
• Audio Tagging by Cross Filtering Noisy Labels
• A Dataset of Dynamic Reverberant Sound Scenes with Directional Interferers for Sound Event Localization and Detection
• Joining Sound Event Detection and Localization Through Spatial Segregation
• Quaternion Convolutional Neural Networks: Current Advances and Future Directions
• On Multitask Loss Function for Audio Event Detection and Localization
• Signal-Aware Direction-of-Arrival Estimation Using Attention Mechanisms
• Joint Analysis of Sound Events and Acoustic Scenes Using Multitask Learning
• Audio scene monitoring using redundant ad-hoc microphone array networks
• ACCDOA: Activity-Coupled Cartesian Direction of Arrival Representation for Sound Event Localization and Detection
• A Review on Sound Source Localization in Robotics: Focusing on Deep Learning Methods
• An Improved Event-Independent Network for Polyphonic Sound Event Localization and Detection
• Leveraging Visual Supervision for Array-based Active Speaker Detection and Localization
• Spectrogram features for audio and speech analysis
• A General Network Architecture for Sound Event Localization and Detection Using Transfer Learning and Recurrent Neural Network
• Spatial mixup: Directional loudness modification as data augmentation for sound event localization and detection
• SSLIDE: Sound Source Localization for Indoors based on Deep Learning
• Spatial LibriSpeech: An Augmented Dataset for Spatial Audio Learning
• L3DAS21 Challenge: Machine Learning for 3D Audio Signal Processing
• First Order Ambisonics Domain Spatial Augmentation for DNN-based Direction of Arrival Estimation
• Diff-SAGe: End-to-End Spatial Audio Generation Using Diffusion Models
• DOA Estimation by DNN-based Denoising and Dereverberation from Sound Intensity Vector
• Sound event localization and detection based on crnn using rectangular filters and channel rotation data augmentation
• Refinement of Direction of Arrival Estimators by Majorization-Minimization Optimization on the Array Manifold
• What Makes Sound Event Localization and Detection Difficult? Insights from Error Analysis
• GISE-51: A scalable isolated sound events dataset
• Sound Event Localization and Detection using Squeeze-Excitation Residual CNNs
• A two-step system for sound event localization and detection
• CNN-based Robust Sound Source Localization with SRP-PHAT for the Extreme Edge
• Active Learning for Sound Event Detection
• SLoClas: A Database for Joint Sound Localization and Classification
• Exploiting Attention-based Sequence-to-Sequence Architectures for Sound Event Localization
• Conditioned Time-Dilated Convolutions for Sound Event Detection
• Single-Microphone Audio Point Source Discriminative Localization From Reverberation Late Tail Estimation
• TASK3 DCASE2021 Challenge: Sound event localization and detection using squeeze-excitation residual CNNs
• Multi-microphone Complex Spectral Mapping for Utterance-wise and Continuous Speech Separation