NWT: Towards natural audio-to-video generation with representation learning

Abstract

In this work we introduce NWT, an expressive speech-to-video model. Unlike approaches that use domain-specific intermediate representations such as pose keypoints, NWT learns its own latent representations, with minimal assumptions about the audio and video content. To this end, we propose a novel discrete variational autoencoder with adversarial loss, dVAE-Adv, which learns a new discrete latent representation we call Memcodes. Memcodes are straightforward to implement, require no additional loss terms, are stable to train compared with other approaches, and show evidence of interpretability. To predict on the Memcode space, we use an autoregressive encoder-decoder model conditioned on audio. Additionally, our model can control latent attributes in the generated video that are not annotated in the data. We train NWT on clips from HBO's Last Week Tonight with John Oliver. NWT consistently scores above other approaches in Mean Opinion Score (MOS) on tests of overall video naturalness, facial naturalness and expressiveness, and lipsync quality. This work sets a strong baseline for generalized audio-to-video synthesis. Samples are available at https://next-week-tonight.github.io/NWT/.

References in corpus (23)

• Auto-Encoding Variational Bayes
• Deep Residual Learning for Image Recognition
• An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
• Categorical Reparameterization with Gumbel-Softmax
• Language Models are Few-Shot Learners
• Neural Discrete Representation Learning
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Zero-Shot Text-to-Image Generation
• A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild
• MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis
• Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis
• Improving Variational Inference with Inverse Autoregressive Flow
• IntroVAE: Introspective Variational Autoencoders for Photographic Image Synthesis
• VideoGPT: Video Generation using VQ-VAE and Transformers
• Adversarial Video Generation on Complex Datasets
• GODIVA: Generating Open-DomaIn Videos from nAtural Descriptions
• Large Memory Layers with Product Keys
• Improving Variational Autoencoder with Deep Feature Consistent and Generative Adversarial Training
• Predicting Video with VQVAE
• Unsupervised Speech Recognition
• Speech2Video Synthesis with 3D Skeleton Regularization and Expressive Body Poses
• PnG BERT: Augmented BERT on Phonemes and Graphemes for Neural TTS
• LumièreNet: Lecture Video Synthesis from Audio