VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency

Abstract

We introduce a new approach for audio-visual speech separation. Given a video, the goal is to extract the speech associated with a face in spite of simultaneous background sounds and/or other human speakers. Whereas existing methods focus on learning the alignment between the speaker's lip movements and the sounds they generate, we propose to leverage the speaker's face appearance as an additional prior to isolate the corresponding vocal qualities they are likely to produce. Our approach jointly learns audio-visual speech separation and cross-modal speaker embeddings from unlabeled video. It yields state-of-the-art results on five benchmark datasets for audio-visual speech separation and enhancement, and generalizes well to challenging real-world videos of diverse scenarios. Our video results and code: http://vision.cs.utexas.edu/projects/VisualVoice/.

In CVPR 2021. Project page: http://vision.cs.utexas.edu/projects/VisualVoice/

References in corpus (11)

• VoxCeleb2: Deep Speaker Recognition
• VoxCeleb: a large-scale speaker identification dataset
• Deep Audio-Visual Speech Recognition
• Spot the conversation: speaker diarisation in the wild
• Perfect match: Improved cross-modal embeddings for audio-visual synchronisation
• Self-Supervised Generation of Spatial Audio for 360 Video
• FaceFilter: Audio-visual speech separation using still images
• Seeing voices and hearing voices: learning discriminative embeddings using cross-modal self-supervision
• On Learning Associations of Faces and Voices
• Disjoint Mapping Network for Cross-modal Matching of Voices and Faces
• Towards Practical Lipreading with Distilled and Efficient Models

Cited by in corpus (3)

• A Survey on Audio Synthesis and Audio-Visual Multimodal Processing
• Learning Audio-Visual Dereverberation
• Improving On-Screen Sound Separation for Open-Domain Videos with Audio-Visual Self-Attention