LEGAN: Disentangled Manipulation of Directional Lighting and Facial Expressions by Leveraging Human Perceptual Judgements

Abstract

Building facial analysis systems that generalize to extreme variations in lighting and facial expressions is a challenging problem that can potentially be alleviated using natural-looking synthetic data. Towards that, we propose LEGAN, a novel synthesis framework that leverages perceptual quality judgments for jointly manipulating lighting and expressions in face images, without requiring paired training data. LEGAN disentangles the lighting and expression subspaces and performs transformations in the feature space before upscaling to the desired output image. The fidelity of the synthetic image is further refined by integrating a perceptual quality estimation model, trained with face images rendered using multiple synthesis methods and their crowd-sourced naturalness ratings, into the LEGAN framework as an auxiliary discriminator. Using objective metrics like FID and LPIPS, LEGAN is shown to generate higher quality face images when compared with popular GAN models like StarGAN and StarGAN-v2 for lighting and expression synthesis. We also conduct a perceptual study using images synthesized by LEGAN and other GAN models and show the correlation between our quality estimation and visual fidelity. Finally, we demonstrate the effectiveness of LEGAN as training data augmenter for expression recognition and face verification tasks.

arXiv commentAMFG Workshop at CVPR 2021 (CVPRW 2021). Get our synthetic face images & their perceptual labels here: https://github.com/Affectiva/LEGAN_Perceptual_Dataset

References in corpus (15)

• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Empirical Evaluation of Rectified Activations in Convolutional Network
• Learning Face Representation from Scratch
• BEGAN: Boundary Equilibrium Generative Adversarial Networks
• Single Image Portrait Relighting
• A Note on the Inception Score
• Strengths and Weaknesses of Deep Learning Models for Face Recognition Against Image Degradations
• VGGFace2: A dataset for recognising faces across pose and age
• First Order Motion Model for Image Animation
• HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Models
• Do We Really Need to Collect Millions of Faces for Effective Face Recognition?
• Bridging the Gap Between Computational Photography and Visual Recognition
• High Fidelity Face Manipulation with Extreme Poses and Expressions
• Approximating Human Judgment of Generated Image Quality