Autonomous Soundscape Augmentation with Multimodal Fusion of Visual and Participant-linked Inputs

Abstract

Autonomous soundscape augmentation systems typically use trained models to pick optimal maskers to effect a desired perceptual change. While acoustic information is paramount to such systems, contextual information, including participant demographics and the visual environment, also influences acoustic perception. Hence, we propose modular modifications to an existing attention-based deep neural network, to allow early, mid-level, and late feature fusion of participant-linked, visual, and acoustic features. Ablation studies on module configurations and corresponding fusion methods using the ARAUS dataset show that contextual features improve the model performance in a statistically significant manner on the normalized ISO Pleasantness, to a mean squared error of $0.1194\pm 0.0012$ for the best-performing all-modality model, against $0.1217\pm 0.0009$ for the audio-only model. Soundscape augmentation systems can thereby leverage multimodal inputs for improved performance. We also investigate the impact of individual participant-linked factors using trained models to illustrate improvements in model explainability.

[v1] 5 pages, 2 figures. Submitted to 2023 IEEE International Conference on Acoustics, Speech and Signal Processing. [v2] 5 pages, 2 figures. Fixed incorrect author list in citation #30

References in corpus (4)

• ARAUS: A Large-Scale Dataset and Baseline Models of Affective Responses to Augmented Urban Soundscapes
• Autonomous In-Situ Soundscape Augmentation via Joint Selection of Masker and Gain
• Deployment of an IoT System for Adaptive In-Situ Soundscape Augmentation
• Squeeze-Excitation Convolutional Recurrent Neural Networks for Audio-Visual Scene Classification

Cited by in corpus (1)

• Anti-noise window: Subjective perception of active noise reduction and effect of informational masking