Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matching

Abstract

Discriminatively localizing sounding objects in cocktail-party, i.e., mixed sound scenes, is commonplace for humans, but still challenging for machines. In this paper, we propose a two-stage learning framework to perform self-supervised class-aware sounding object localization. First, we propose to learn robust object representations by aggregating the candidate sound localization results in the single source scenes. Then, class-aware object localization maps are generated in the cocktail-party scenarios by referring the pre-learned object knowledge, and the sounding objects are accordingly selected by matching audio and visual object category distributions, where the audiovisual consistency is viewed as the self-supervised signal. Experimental results in both realistic and synthesized cocktail-party videos demonstrate that our model is superior in filtering out silent objects and pointing out the location of sounding objects of different classes. Code is available at https://github.com/DTaoo/Discriminative-Sounding-Objects-Localization.

arXiv commentTo appear in NeurIPS 2020. Previous Title: Learning to Discriminatively Localize Sounding Objects in a Cocktail-party Scenario

References in corpus (5)

• Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks
• Self-Supervised Learning by Cross-Modal Audio-Video Clustering
• Curriculum Audiovisual Learning
• Multiple Sound Sources Localization from Coarse to Fine
• Cross-Task Transfer for Geotagged Audiovisual Aerial Scene Recognition

Cited by in corpus (4)

• Exploiting Transformation Invariance and Equivariance for Self-supervised Sound Localisation
• Induction Network: Audio-Visual Modality Gap-Bridging for Self-Supervised Sound Source Localization
• Deep Video Inpainting Guided by Audio-Visual Self-Supervision
• Dual Normalization Multitasking for Audio-Visual Sounding Object Localization