Attention-based cross-modal fusion for audio-visual voice activity detection in musical video streams

Abstract

Many previous audio-visual voice-related works focus on speech, ignoring the singing voice in the growing number of musical video streams on the Internet. For processing diverse musical video data, voice activity detection is a necessary step. This paper attempts to detect the speech and singing voices of target performers in musical video streams using audiovisual information. To integrate information of audio and visual modalities, a multi-branch network is proposed to learn audio and image representations, and the representations are fused by attention based on semantic similarity to shape the acoustic representations through the probability of anchor vocalization. Experiments show the proposed audio-visual multi-branch network far outperforms the audio-only model in challenging acoustic environments, indicating the cross-modal information fusion based on semantic correlation is sensible and successful.

arXiv commentAccepted by INTERSPEECH 2021

References in corpus (3)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• End-to-end Audiovisual Speech Activity Detection with Bimodal Recurrent Neural Models
• Sound Event Detection with Sequentially Labelled Data Based on Connectionist Temporal Classification and Unsupervised Clustering