LipSound2: Self-Supervised Pre-Training for Lip-to-Speech Reconstruction and Lip Reading

Abstract

The aim of this work is to investigate the impact of crossmodal self-supervised pre-training for speech reconstruction (video-to-audio) by leveraging the natural co-occurrence of audio and visual streams in videos. We propose LipSound2 which consists of an encoder-decoder architecture and location-aware attention mechanism to map face image sequences to mel-scale spectrograms directly without requiring any human annotations. The proposed LipSound2 model is firstly pre-trained on $\sim$2400h multi-lingual (e.g. English and German) audio-visual data (VoxCeleb2). To verify the generalizability of the proposed method, we then fine-tune the pre-trained model on domain-specific datasets (GRID, TCD-TIMIT) for English speech reconstruction and achieve a significant improvement on speech quality and intelligibility compared to previous approaches in speaker-dependent and -independent settings. In addition to English, we conduct Chinese speech reconstruction on the CMLR dataset to verify the impact on transferability. Lastly, we train the cascaded lip reading (video-to-text) system by fine-tuning the generated audios on a pre-trained speech recognition system and achieve state-of-the-art performance on both English and Chinese benchmark datasets.

ACCEPTED IN IEEE Transactions on Neural Networks and Learning Systems

References in corpus (7)

• WaveNet: A Generative Model for Raw Audio
• Attention-Based Models for Speech Recognition
• BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
• FaceFilter: Audio-visual speech separation using still images
• Learning Spatio-Temporal Features with Two-Stream Deep 3D CNNs for Lipreading
• Visual Features for Context-Aware Speech Recognition
• DualLip: A System for Joint Lip Reading and Generation