Visually-Guided Sound Source Separation with Audio-Visual Predictive Coding
arXiv:2306.10684 · doi:10.1109/TNNLS.2023.3288022
Abstract
The framework of visually-guided sound source separation generally consists of three parts: visual feature extraction, multimodal feature fusion, and sound signal processing. An ongoing trend in this field has been to tailor involved visual feature extractor for informative visual guidance and separately devise module for feature fusion, while utilizing U-Net by default for sound analysis. However, such divide-and-conquer paradigm is parameter inefficient and, meanwhile, may obtain suboptimal performance as jointly optimizing and harmonizing various model components is challengeable. By contrast, this paper presents a novel approach, dubbed audio-visual predictive coding (AVPC), to tackle this task in a parameter efficient and more effective manner. The network of AVPC features a simple ResNet-based video analysis network for deriving semantic visual features, and a predictive coding-based sound separation network that can extract audio features, fuse multimodal information, and predict sound separation masks in the same architecture. By iteratively minimizing the prediction error between features, AVPC integrates audio and visual information recursively, leading to progressively improved performance. In addition, we develop a valid self-supervised learning strategy for AVPC via co-predicting two audio-visual representations of the same sound source. Extensive evaluations demonstrate that AVPC outperforms several baselines in separating musical instrument sounds, while reducing the model size significantly. Code is available at: https://github.com/zjsong/Audio-Visual-Predictive-Coding.
arXiv commentAccepted to IEEE Transactions on Neural Networks and Learning Systems (T-NNLS)
References in corpus (16)
- Decoupled Weight Decay Regularization
- Bootstrap your own latent: A new approach to self-supervised Learning
- Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation
- Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separation
- Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization
- Self-Supervised Learning by Cross-Modal Audio-Video Clustering
- Self-Supervised MultiModal Versatile Networks
- Creating A Multi-track Classical Musical Performance Dataset for Multimodal Music Analysis: Challenges, Insights, and Applications
- Face Landmark-based Speaker-Independent Audio-Visual Speech Enhancement in Multi-Talker Environments
- Deep Predictive Coding Network for Object Recognition
- Curriculum Audiovisual Learning
- Self-Supervised Predictive Learning: A Negative-Free Method for Sound Source Localization in Visual Scenes
- Towards Versatile Embodied Navigation
- Learning Audio-Visual Dynamics Using Scene Graphs for Audio Source Separation
- Learning Controllable Disentangled Representations with Decorrelation Regularization
- Toward a Controllable Disentanglement Network