Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separation

Abstract

We present a joint audio-visual model for isolating a single speech signal from a mixture of sounds such as other speakers and background noise. Solving this task using only audio as input is extremely challenging and does not provide an association of the separated speech signals with speakers in the video. In this paper, we present a deep network-based model that incorporates both visual and auditory signals to solve this task. The visual features are used to "focus" the audio on desired speakers in a scene and to improve the speech separation quality. To train our joint audio-visual model, we introduce AVSpeech, a new dataset comprised of thousands of hours of video segments from the Web. We demonstrate the applicability of our method to classic speech separation tasks, as well as real-world scenarios involving heated interviews, noisy bars, and screaming children, only requiring the user to specify the face of the person in the video whose speech they want to isolate. Our method shows clear advantage over state-of-the-art audio-only speech separation in cases of mixed speech. In addition, our model, which is speaker-independent (trained once, applicable to any speaker), produces better results than recent audio-visual speech separation methods that are speaker-dependent (require training a separate model for each speaker of interest).

arXiv commentAccepted to SIGGRAPH 2018. Project webpage: https://looking-to-listen.github.io

References in corpus (6)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Object Detectors Emerge in Deep Scene CNNs
• MOON: A Mixed Objective Optimization Network for the Recognition of Facial Attributes
• Deep Multimodal Speaker Naming
• Putting a Face to the Voice: Fusing Audio and Visual Signals Across a Video to Determine Speakers
• Audio-Visual Speech Enhancement Using Multimodal Deep Convolutional Neural Networks

Cited by in corpus (37)

• Multimodal Intelligence: Representation Learning, Information Fusion, and Applications
• Self-Supervised Speech Representation Learning: A Review
• Visual Speech Recognition for Multiple Languages in the Wild
• Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection
• Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels
• Neural Target Speech Extraction: An Overview
• GraphMFT: A Graph Network based Multimodal Fusion Technique for Emotion Recognition in Conversation
• Audio-Visual Speech Separation and Dereverberation with a Two-Stage Multimodal Network
• Face Landmark-based Speaker-Independent Audio-Visual Speech Enhancement in Multi-Talker Environments
• Deep-Learning-Based Audio-Visual Speech Enhancement in Presence of Lombard Effect
• UniCon: Unified Context Network for Robust Active Speaker Detection
• Complex Neural Spatial Filter: Enhancing Multi-channel Target Speech Separation in Complex Domain
• Towards Generating Ambisonics Using Audio-Visual Cue for Virtual Reality
• Speaker Extraction with Co-Speech Gestures Cue
• Does Visual Self-Supervision Improve Learning of Speech Representations for Emotion Recognition?
• Conditioned Source Separation for Music Instrument Performances
• Look Once to Hear: Target Speech Hearing with Noisy Examples
• Latent-Domain Predictive Neural Speech Coding
• Families In Wild Multimedia: A Multimodal Database for Recognizing Kinship
• On Training Targets and Objective Functions for Deep-Learning-Based Audio-Visual Speech Enhancement
• Lip-to-Speech Synthesis for Arbitrary Speakers in the Wild
• Improving performance of real-time full-band blind packet-loss concealment with predictive network
• PEANUT: A Human-AI Collaborative Tool for Annotating Audio-Visual Data
• Points2Sound: From mono to binaural audio using 3D point cloud scenes
• Leveraging Visual Supervision for Array-based Active Speaker Detection and Localization
• Visually-Guided Sound Source Separation with Audio-Visual Predictive Coding
• Universal Speech Token Learning via Low-Bitrate Neural Codec and Pretrained Representations
• Bio-Inspired Modality Fusion for Active Speaker Detection
• Effects of Lombard Reflex on the Performance of Deep-Learning-Based Audio-Visual Speech Enhancement Systems
• Uncovering the Visual Contribution in Audio-Visual Speech Recognition
• TDFNet: An Efficient Audio-Visual Speech Separation Model with Top-down Fusion
• RAVSS: Robust Audio-Visual Speech Separation in Multi-Speaker Scenarios with Missing Visual Cues
• Visual-Informed Speech Enhancement Using Attention-Based Beamforming
• Extreme-scale Talking-Face Video Upsampling with Audio-Visual Priors
• Deep Video Inpainting Guided by Audio-Visual Self-Supervision
• An Empirical Study of Visual Features for DNN based Audio-Visual Speech Enhancement in Multi-talker Environments
• Face and Voice Cross-modal Association with Learning Convex Feature Embedding