A Neural Virtual Anchor Synthesizer based on Seq2Seq and GAN Models

Abstract

This paper presents a novel framework to generate realistic face video of an anchor, who is reading certain news. This task is also known as Virtual Anchor. Given some paragraphs of words, we first utilize a pretrained Word2Vec model to embed each word into a vector; then we utilize a Seq2Seq-based model to translate these word embeddings into action units and head poses of the target anchor; these action units and head poses will be concatenated with facial landmarks as well as the former $n$ synthesized frames, and the concatenation serves as input of a Pix2PixHD-based model to synthesize realistic facial images for the virtual anchor. The experimental results demonstrate our framework is feasible for the synthesis of virtual anchor.

Accepted to ISMAR 2019

References in corpus (11)

• Efficient Estimation of Word Representations in Vector Space
• Sequence to Sequence Learning with Neural Networks
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Convolutional Sequence to Sequence Learning
• Regularizing and Optimizing LSTM Language Models
• Automatic Face Reenactment
• Towards the Automatic Anime Characters Creation with Generative Adversarial Networks
• Multiplicative LSTM for sequence modelling
• Face Transfer with Generative Adversarial Network
• CycleGAN Face-off
• Video synthesis of human upper body with realistic face

Cited by in corpus (3)

• A Realistic Face-to-Face Conversation System based on Deep Neural Networks
• Facial Pose Estimation by Deep Learning from Label Distributions
• How Old Are You? Face Age Translation with Identity Preservation Using GANs