Deep Audio-Visual Learning: A Survey

Abstract

Audio-visual learning, aimed at exploiting the relationship between audio and visual modalities, has drawn considerable attention since deep learning started to be used successfully. Researchers tend to leverage these two modalities either to improve the performance of previously considered single-modality tasks or to address new challenging problems. In this paper, we provide a comprehensive survey of recent audio-visual learning development. We divide the current audio-visual learning tasks into four different subfields: audio-visual separation and localization, audio-visual correspondence learning, audio-visual generation, and audio-visual representation learning. State-of-the-art methods as well as the remaining challenges of each subfield are further discussed. Finally, we summarize the commonly used datasets and performance metrics.

References in corpus (22)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Two-Stream Convolutional Networks for Action Recognition in Videos
• The Kinetics Human Action Video Dataset
• VoxCeleb2: Deep Speaker Recognition
• VoxCeleb: a large-scale speaker identification dataset
• YouTube-8M: A Large-Scale Video Classification Benchmark
• Deep Audio-Visual Speech Recognition
• A Short Note on the Kinetics-700-2020 Human Action Dataset
• A Short Note about Kinetics-600
• Speaker-independent Speech Separation with Deep Attractor Network
• SEWA DB: A Rich Database for Audio-Visual Emotion and Sentiment Research in the Wild
• Perfect match: Improved cross-modal embeddings for audio-visual synchronisation
• ObamaNet: Photo-realistic lip-sync from text
• See, Hear, and Read: Deep Aligned Representations
• Face Landmark-based Speaker-Independent Audio-Visual Speech Enhancement in Multi-Talker Environments
• Generative Choreography using Deep Learning
• Putting a Face to the Voice: Fusing Audio and Visual Signals Across a Video to Determine Speakers
• Arbitrary Talking Face Generation via Attentional Audio-Visual Coherence Learning
• Speech-Driven Facial Reenactment Using Conditional Generative Adversarial Networks
• Vision-based Detection of Acoustic Timed Events: a Case Study on Clarinet Note Onsets
• CMCGAN: A Uniform Framework for Cross-Modal Visual-Audio Mutual Generation
• Towards Audio to Scene Image Synthesis using Generative Adversarial Network

Cited by in corpus (5)

• Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation
• Sep-Stereo: Visually Guided Stereophonic Audio Generation by Associating Source Separation
• Ear2Face: Deep Biometric Modality Mapping
• Automatic Face Understanding: Recognizing Families in Photos
• AudioVisual Video Summarization