Weakly Supervised Representation Learning for Unsynchronized Audio-Visual Events

Abstract

Audio-visual representation learning is an important task from the perspective of designing machines with the ability to understand complex events. To this end, we propose a novel multimodal framework that instantiates multiple instance learning. We show that the learnt representations are useful for classifying events and localizing their characteristic audio-visual elements. The system is trained using only video-level event labels without any timing information. An important feature of our method is its capacity to learn from unsynchronized audio-visual events. We achieve state-of-the-art results on a large-scale dataset of weakly-labeled audio event videos. Visualizations of localized visual regions and audio segments substantiate our system's efficacy, especially when dealing with noisy situations where modality-specific cues appear asynchronously.

References in corpus (9)

• Adam: A Method for Stochastic Optimization
• YouTube-8M: A Large-Scale Video Classification Benchmark
• SoundNet: Learning Sound Representations from Unlabeled Video
• Audio Event Detection using Weakly Labeled Data
• Contextual Action Recognition with R*CNN
• See, Hear, and Read: Deep Aligned Representations
• How good are detection proposals, really?
• ContextLocNet: Context-Aware Deep Network Models for Weakly Supervised Localization
• Surrey-cvssp system for DCASE2017 challenge task4

Cited by in corpus (3)

• Audiovisual Transformer Architectures for Large-Scale Classification and Synchronization of Weakly Labeled Audio Events
• Real-Time Idling Vehicles Detection using Combined Audio-Visual Deep Learning
• Audiovisual transfer learning for audio tagging and sound event detection