Improving Generative Adversarial Network Generalization for Facial Expression Synthesis

Abstract

Facial expression synthesis aims to generate realistic facial expressions while preserving identity. Existing conditional generative adversarial networks (GANs) achieve excellent image-to-image translation results, but their performance often degrades when test images differ from the training dataset. We present Regression GAN (RegGAN), a model that learns an intermediate representation to improve generalization beyond the training distribution. RegGAN consists of two components: a regression layer with local receptive fields that learns expression details by minimizing the reconstruction error through a ridge regression loss, and a refinement network trained adversarially to enhance the realism of generated images. We train RegGAN on the CFEE dataset and evaluate its generalization performance both on CFEE and challenging out-of-distribution images, including celebrity photos, portraits, statues, and avatar renderings. For evaluation, we employ four widely used metrics: Expression Classification Score (ECS) for expression quality, Face Similarity Score (FSS) for identity preservation, QualiCLIP for perceptual realism, and Fréchet Inception Distance (FID) for assessing both expression quality and realism. RegGAN outperforms six state-of-the-art models in ECS, FID, and QualiCLIP, while ranking second in FSS. Human evaluations indicate that RegGAN surpasses the best competing model by 25% in expression quality, 26% in identity preservation, and 30% in realism.

References in corpus (8)

• How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)
• Learning Spatial Attention for Face Super-Resolution
• Scene recognition with CNNs: objects, scales and dataset bias
• Edge and Identity Preserving Network for Face Super-Resolution
• GP-GAN: Gender Preserving GAN for Synthesizing Faces from Landmarks
• SARGAN: Spatial Attention-based Residuals for Facial Expression Manipulation
• US-GAN: On the importance of Ultimate Skip Connection for Facial Expression Synthesis
• Masked Linear Regression for Learning Local Receptive Fields for Facial Expression Synthesis