UAVM: Towards Unifying Audio and Visual Models

Abstract

Conventional audio-visual models have independent audio and video branches. In this work, we unify the audio and visual branches by designing a Unified Audio-Visual Model (UAVM). The UAVM achieves a new state-of-the-art audio-visual event classification accuracy of 65.8% on VGGSound. More interestingly, we also find a few intriguing properties of UAVM that the modality-independent counterparts do not have.

Published in Signal Processing Letters. Code at https://github.com/YuanGongND/uavm

Cited by in corpus (3)

• Multimodal Fish Feeding Intensity Assessment in Aquaculture
• Versatile audio-visual learning for emotion recognition
• USpeech: Ultrasound-Enhanced Speech with Minimal Human Effort via Cross-Modal Synthesis