FastLTS: Non-Autoregressive End-to-End Unconstrained Lip-to-Speech Synthesis

Abstract

Unconstrained lip-to-speech synthesis aims to generate corresponding speeches from silent videos of talking faces with no restriction on head poses or vocabulary. Current works mainly use sequence-to-sequence models to solve this problem, either in an autoregressive architecture or a flow-based non-autoregressive architecture. However, these models suffer from several drawbacks: 1) Instead of directly generating audios, they use a two-stage pipeline that first generates mel-spectrograms and then reconstructs audios from the spectrograms. This causes cumbersome deployment and degradation of speech quality due to error propagation; 2) The audio reconstruction algorithm used by these models limits the inference speed and audio quality, while neural vocoders are not available for these models since their output spectrograms are not accurate enough; 3) The autoregressive model suffers from high inference latency, while the flow-based model has high memory occupancy: neither of them is efficient enough in both time and memory usage. To tackle these problems, we propose FastLTS, a non-autoregressive end-to-end model which can directly synthesize high-quality speech audios from unconstrained talking videos with low latency, and has a relatively small model size. Besides, different from the widely used 3D-CNN visual frontend for lip movement encoding, we for the first time propose a transformer-based visual frontend for this task. Experiments show that our model achieves $19.76\times$ speedup for audio waveform generation compared with the current autoregressive model on input sequences of 3 seconds, and obtains superior audio quality.

10 pages, 5 figures, accepted by ACMMM 2022

References in corpus (10)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Sequence to Sequence Learning with Neural Networks
• Is Space-Time Attention All You Need for Video Understanding?
• HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis
• FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
• A Survey on Neural Speech Synthesis
• An Image is Worth 16x16 Words, What is a Video Worth?
• End-to-End Video-To-Speech Synthesis using Generative Adversarial Networks
• Lip to Speech Synthesis with Visual Context Attentional GAN
• Transformer-Based Video Front-Ends for Audio-Visual Speech Recognition for Single and Multi-Person Video

Cited by in corpus (1)

• UMMAFormer: A Universal Multimodal-adaptive Transformer Framework for Temporal Forgery Localization