Recent Advances and Challenges in Deep Audio-Visual Correlation Learning

Abstract

Audio-visual correlation learning aims to capture essential correspondences and understand natural phenomena between audio and video. With the rapid growth of deep learning, an increasing amount of attention has been paid to this emerging research issue. Through the past few years, various methods and datasets have been proposed for audio-visual correlation learning, which motivate us to conclude a comprehensive survey. This survey paper focuses on state-of-the-art (SOTA) models used to learn correlations between audio and video, but also discusses some tasks of definition and paradigm applied in AI multimedia. In addition, we investigate some objective functions frequently used for optimizing audio-visual correlation learning models and discuss how audio-visual data is exploited in the optimization process. Most importantly, we provide an extensive comparison and summarization of the recent progress of SOTA audio-visual correlation learning and discuss future research directions.

8 pages, 1 figure

References in corpus (11)

• Deep Learning for Audio Signal Processing
• Efficient Training of Audio Transformers with Patchout
• TCLR: Temporal Contrastive Learning for Video Representation
• Self-supervised Video Representation Learning Using Inter-intra Contrastive Framework
• Look, Listen, and Attend: Co-Attention Network for Self-Supervised Audio-Visual Representation Learning
• Exploiting Image-trained CNN Architectures for Unconstrained Video Classification
• ERANNs: Efficient Residual Audio Neural Networks for Audio Pattern Recognition
• End-to-End Video-To-Speech Synthesis using Generative Adversarial Networks
• UAVM: Towards Unifying Audio and Visual Models
• Similarity Contrastive Estimation for Image and Video Soft Contrastive Self-Supervised Learning
• Self-supervised object detection from audio-visual correspondence