A Four-Stage Data Augmentation Approach to ResNet-Conformer Based Acoustic Modeling for Sound Event Localization and Detection

Abstract

In this paper, we propose a novel four-stage data augmentation approach to ResNet-Conformer based acoustic modeling for sound event localization and detection (SELD). First, we explore two spatial augmentation techniques, namely audio channel swapping (ACS) and multi-channel simulation (MCS), to deal with data sparsity in SELD. ACS and MDS focus on augmenting the limited training data with expanding direction of arrival (DOA) representations such that the acoustic models trained with the augmented data are robust to localization variations of acoustic sources. Next, time-domain mixing (TDM) and time-frequency masking (TFM) are also investigated to deal with overlapping sound events and data diversity. Finally, ACS, MCS, TDM and TFM are combined in a step-by-step manner to form an effective four-stage data augmentation scheme. Tested on the Detection and Classification of Acoustic Scenes and Events (DCASE) 2020 data set, our proposed augmentation approach greatly improves the system performance, ranking our submitted system in the first place in the SELD task of the DCASE 2020 Challenge. Furthermore, we employ a ResNet-Conformer architecture to model both global and local context dependencies of an audio sequence and win the first place in the DCASE 2022 SELD evaluations.

arXiv comment13 pages, 8 figures, Accepted by Transactions on Audio, Speech and Language Processing

References in corpus (14)

• Adam: A Method for Stochastic Optimization
• mixup: Beyond Empirical Risk Minimization
• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• The Effectiveness of Data Augmentation in Image Classification using Deep Learning
• Convolutional Recurrent Neural Networks for Polyphonic Sound Event Detection
• Sound Event Localization and Detection of Overlapping Sources Using Convolutional Recurrent Neural Networks
• Conformer: Convolution-augmented Transformer for Speech Recognition
• Recurrent Neural Networks for Polyphonic Sound Event Detection in Real Life Recordings
• Polyphonic Sound Event Detection and Localization using a Two-Stage Strategy
• Deep Convolutional Neural Networks and Data Augmentation for Acoustic Event Detection
• Polyphonic Sound Event Detection by using Capsule Neural Networks
• A Dataset of Reverberant Spatial Sound Scenes with Moving Sources for Sound Event Localization and Detection
• Sound Event Localization and Detection Using Activity-Coupled Cartesian DOA Vector and RD3net
• Continuous Speech Separation with Conformer

Cited by in corpus (6)

• A Survey of Sound Source Localization with Deep Learning Methods
• SALSA: Spatial Cue-Augmented Log-Spectrogram Features for Polyphonic Sound Event Localization and Detection
• SALSA-Lite: A Fast and Effective Feature for Polyphonic Sound Event Localization and Detection with Microphone Arrays
• A Dataset of Dynamic Reverberant Sound Scenes with Directional Interferers for Sound Event Localization and Detection
• Ensemble of ACCDOA- and EINV2-based Systems with D3Nets and Impulse Response Simulation for Sound Event Localization and Detection
• Spatial mixup: Directional loudness modification as data augmentation for sound event localization and detection