Learning Audio-Visual Dereverberation

Abstract

Reverberation not only degrades the quality of speech for human perception, but also severely impacts the accuracy of automatic speech recognition. Prior work attempts to remove reverberation based on the audio modality only. Our idea is to learn to dereverberate speech from audio-visual observations. The visual environment surrounding a human speaker reveals important cues about the room geometry, materials, and speaker location, all of which influence the precise reverberation effects. We introduce Visually-Informed Dereverberation of Audio (VIDA), an end-to-end approach that learns to remove reverberation based on both the observed monaural sound and visual scene. In support of this new task, we develop a large-scale dataset SoundSpaces-Speech that uses realistic acoustic renderings of speech in real-world 3D scans of homes offering a variety of room acoustics. Demonstrating our approach on both simulated and real imagery for speech enhancement, speech recognition, and speaker identification, we show it achieves state-of-the-art performance and substantially improves over audio-only methods.

arXiv commentAccepted at ICASSP 2023. This is the longer version of the five-page camera-ready paper. Project page: https://vision.cs.utexas.edu/projects/learning-audio-visual-dereverberation

References in corpus (12)

• VoxCeleb: a large-scale speaker identification dataset
• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification
• The Replica Dataset: A Digital Replica of Indoor Spaces
• Matterport3D: Learning from RGB-D Data in Indoor Environments
• Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization
• Building and Evaluation of a Real Room Impulse Response Dataset
• Self-Supervised Generation of Spatial Audio for 360 Video
• Audio-Visual Speech Separation and Dereverberation with a Two-Stage Multimodal Network
• Learning to Separate Object Sounds by Watching Unlabeled Video
• VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency
• VisualEchoes: Spatial Image Representation Learning through Echolocation