Controlled AutoEncoders to Generate Faces from Voices

Abstract

Multiple studies in the past have shown that there is a strong correlation between human vocal characteristics and facial features. However, existing approaches generate faces simply from voice, without exploring the set of features that contribute to these observed correlations. A computational methodology to explore this can be devised by rephrasing the question to: "how much would a target face have to change in order to be perceived as the originator of a source voice?" With this in perspective, we propose a framework to morph a target face in response to a given voice in a way that facial features are implicitly guided by learned voice-face correlation in this paper. Our framework includes a guided autoencoder that converts one face to another, controlled by a unique model-conditioning component called a gating controller which modifies the reconstructed face based on input voice recordings. We evaluate the framework on VoxCelab and VGGFace datasets through human subjects and face retrieval. Various experiments demonstrate the effectiveness of our proposed model.

References in corpus (8)

• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
• Identification of Deep Network Generated Images Using Disparities in Color Components
• GAN Dissection: Visualizing and Understanding Generative Adversarial Networks
• Adversarial Ranking for Language Generation
• HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Models
• On Learning Associations of Faces and Voices
• X2Face: A network for controlling face generation by using images, audio, and pose codes