Every Smile is Unique: Landmark-Guided Diverse Smile Generation

Abstract

Each smile is unique: one person surely smiles in different ways (e.g., closing/opening the eyes or mouth). Given one input image of a neutral face, can we generate multiple smile videos with distinctive characteristics? To tackle this one-to-many video generation problem, we propose a novel deep learning architecture named Conditional Multi-Mode Network (CMM-Net). To better encode the dynamics of facial expressions, CMM-Net explicitly exploits facial landmarks for generating smile sequences. Specifically, a variational auto-encoder is used to learn a facial landmark embedding. This single embedding is then exploited by a conditional recurrent network which generates a landmark embedding sequence conditioned on a specific expression (e.g., spontaneous smile). Next, the generated landmark embeddings are fed into a multi-mode recurrent landmark generator, producing a set of landmark sequences still associated to the given smile class but clearly distinct from each other. Finally, these landmark sequences are translated into face videos. Our experimental results demonstrate the effectiveness of our CMM-Net in generating realistic videos of multiple smile expressions.

arXiv commentAccepted as a poster in Conference on Computer Vision and Pattern Recognition (CVPR), 2018

References in corpus (10)

• Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
• Improved Techniques for Training GANs
• DRAW: A Recurrent Neural Network For Image Generation
• Generating Videos with Scene Dynamics
• Deep Convolutional Inverse Graphics Network
• Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders
• Markov Chain Monte Carlo and Variational Inference: Bridging the Gap
• Learning to Generate Long-term Future via Hierarchical Prediction
• Generative Visual Manipulation on the Natural Image Manifold
• Nonparametric Variational Auto-encoders for Hierarchical Representation Learning

Cited by in corpus (6)

• Dynamic Facial Expression Generation on Hilbert Hypersphere with Conditional Wasserstein Generative Adversarial Nets
• Triple consistency loss for pairing distributions in GAN-based face synthesis
• TransGaGa: Geometry-Aware Unsupervised Image-to-Image Translation
• DwNet: Dense warp-based network for pose-guided human video generation
• Cycle In Cycle Generative Adversarial Networks for Keypoint-Guided Image Generation
• GestureGAN for Hand Gesture-to-Gesture Translation in the Wild