Improving On-Screen Sound Separation for Open-Domain Videos with Audio-Visual Self-Attention
arXiv:2106.09669
Abstract
We introduce a state-of-the-art audio-visual on-screen sound separation system which is capable of learning to separate sounds and associate them with on-screen objects by looking at in-the-wild videos. We identify limitations of previous work on audio-visual on-screen sound separation, including the simplicity and coarse resolution of spatio-temporal attention, and poor convergence of the audio separation model. Our proposed model addresses these issues using cross-modal and self-attention modules that capture audio-visual dependencies at a finer resolution over time, and by unsupervised pre-training of audio separation model. These improvements allow the model to generalize to a much wider set of unseen videos. We also show a robust way to further improve the generalization capability of our models by calibrating the probabilities of our audio-visual on-screen classifier, using only a small amount of in-domain videos labeled for their on-screen presence. For evaluation and semi-supervised training, we collected human annotations of on-screen audio from a large database of in-the-wild videos (YFCC100m). Our results show marked improvements in on-screen separation performance, in more general conditions than previous methods.
References in corpus (12)
- Scikit-learn: Machine Learning in Python
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- YFCC100M: The New Data in Multimedia Research
- Is Space-Time Attention All You Need for Video Understanding?
- Improving Universal Sound Separation Using Sound Classification
- Two-Step Sound Source Separation: Training on Learned Latent Targets
- Unsupervised Deep Clustering for Source Separation: Direct Learning from Mixtures using Spatial Information
- VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency
- VidTr: Video Transformer Without Convolutions
- Listen to What You Want: Neural Network-based Universal Sound Selector
- Unsupervised Sound Localization via Iterative Contrastive Learning
- MPN: Multimodal Parallel Network for Audio-Visual Event Localization