Lip-to-Speech Synthesis for Arbitrary Speakers in the Wild

Abstract

In this work, we address the problem of generating speech from silent lip videos for any speaker in the wild. In stark contrast to previous works, our method (i) is not restricted to a fixed number of speakers, (ii) does not explicitly impose constraints on the domain or the vocabulary and (iii) deals with videos that are recorded in the wild as opposed to within laboratory settings. The task presents a host of challenges, with the key one being that many features of the desired target speech, like voice, pitch and linguistic content, cannot be entirely inferred from the silent face video. In order to handle these stochastic variations, we propose a new VAE-GAN architecture that learns to associate the lip and speech sequences amidst the variations. With the help of multiple powerful discriminators that guide the training process, our generator learns to synthesize speech sequences in any voice for the lip movements of any person. Extensive experiments on multiple datasets show that we outperform all baselines by a large margin. Further, our network can be fine-tuned on videos of specific identities to achieve a performance comparable to single-speaker models that are trained on $4\times$ more data. We conduct numerous ablation studies to analyze the effect of different modules of our architecture. We also provide a demo video that demonstrates several qualitative results along with the code and trained models on our website: \url{http://cvit.iiit.ac.in/research/projects/cvit-projects/lip-to-speech-synthesis}}

arXiv commentAccepted in ACM-MM 2022, 9 pages, 2 pages supplementary, 7 Figures

References in corpus (15)

• Auto-Encoding Variational Bayes
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• A Simple Framework for Contrastive Learning of Visual Representations
• Representation Learning with Contrastive Predictive Coding
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• Robust Real-World Image Super-Resolution against Adversarial Attacks
• A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild
• Deep Audio-Visual Speech Recognition
• Lip Reading Sentences in the Wild
• Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separation
• Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis
• Variational Approaches for Auto-Encoding Generative Adversarial Networks
• Perfect match: Improved cross-modal embeddings for audio-visual synchronisation
• ObamaNet: Photo-realistic lip-sync from text
• Harnessing AI for Speech Reconstruction using Multi-view Silent Video Feed

Cited by in corpus (2)

• UMMAFormer: A Universal Multimodal-adaptive Transformer Framework for Temporal Forgery Localization
• Towards Accurate Lip-to-Speech Synthesis in-the-Wild