Learning Representations from Audio-Visual Spatial Alignment

Abstract

We introduce a novel self-supervised pretext task for learning representations from audio-visual content. Prior work on audio-visual representation learning leverages correspondences at the video level. Approaches based on audio-visual correspondence (AVC) predict whether audio and video clips originate from the same or different video instances. Audio-visual temporal synchronization (AVTS) further discriminates negative pairs originated from the same video instance but at different moments in time. While these approaches learn high-quality representations for downstream tasks such as action recognition, their training objectives disregard spatial cues naturally occurring in audio and visual signals. To learn from these spatial cues, we tasked a network to perform contrastive audio-visual spatial alignment of 360° video and spatial audio. The ability to perform spatial alignment is enhanced by reasoning over the full spatial content of the 360° video using a transformer architecture to combine representations from multiple viewpoints. The advantages of the proposed pretext task are demonstrated on a variety of audio and visual downstream tasks, including audio-visual correspondence, spatial alignment, action recognition, and video semantic segmentation.

To appear at Advances in Neural Information Processing Systems (NeurIPS), 2020

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• A Simple Framework for Contrastive Learning of Visual Representations
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• The Kinetics Human Action Video Dataset
• Momentum Contrast for Unsupervised Visual Representation Learning
• Self-Supervised Learning by Cross-Modal Audio-Video Clustering
• SoundNet: Learning Sound Representations from Unlabeled Video
• Self-Supervised Generation of Spatial Audio for 360 Video
• Contrastive Learning with Adversarial Examples
• Self-Supervised Relative Depth Learning for Urban Scene Understanding
• Music Gesture for Visual Sound Separation

Cited by in corpus (3)

• Unsupervised Sound Localization via Iterative Contrastive Learning
• Learning Cross-modal Contrastive Features for Video Domain Adaptation
• Language-Mediated, Object-Centric Representation Learning