Deep Audio-Visual Learning: A Survey
arXiv:2001.04758
Abstract
Audio-visual learning, aimed at exploiting the relationship between audio and visual modalities, has drawn considerable attention since deep learning started to be used successfully. Researchers tend to leverage these two modalities either to improve the performance of previously considered single-modality tasks or to address new challenging problems. In this paper, we provide a comprehensive survey of recent audio-visual learning development. We divide the current audio-visual learning tasks into four different subfields: audio-visual separation and localization, audio-visual correspondence learning, audio-visual generation, and audio-visual representation learning. State-of-the-art methods as well as the remaining challenges of each subfield are further discussed. Finally, we summarize the commonly used datasets and performance metrics.
References in corpus (9)
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Two-Stream Convolutional Networks for Action Recognition in Videos
- The Kinetics Human Action Video Dataset
- YouTube-8M: A Large-Scale Video Classification Benchmark
- ObamaNet: Photo-realistic lip-sync from text
- See, Hear, and Read: Deep Aligned Representations
- Putting a Face to the Voice: Fusing Audio and Visual Signals Across a Video to Determine Speakers
- Vision-based Detection of Acoustic Timed Events: a Case Study on Clarinet Note Onsets
- CMCGAN: A Uniform Framework for Cross-Modal Visual-Audio Mutual Generation