SARGAN: Spatial Attention-based Residuals for Facial Expression Manipulation

Abstract

Encoder-decoder based architecture has been widely used in the generator of generative adversarial networks for facial manipulation. However, we observe that the current architecture fails to recover the input image color, rich facial details such as skin color or texture and introduces artifacts as well. In this paper, we present a novel method named SARGAN that addresses the above-mentioned limitations from three perspectives. First, we employed spatial attention-based residual block instead of vanilla residual blocks to properly capture the expression-related features to be changed while keeping the other features unchanged. Second, we exploited a symmetric encoder-decoder network to attend facial features at multiple scales. Third, we proposed to train the complete network with a residual connection which relieves the generator of pressure to generate the input face image thereby producing the desired expression by directly feeding the input image towards the end of the generator. Both qualitative and quantitative experimental results show that our proposed model performs significantly better than state-of-the-art methods. In addition, existing models require much larger datasets for training but their performance degrades on out-of-distribution images. In contrast, SARGAN can be trained on smaller facial expressions datasets, which generalizes well on out-of-distribution images including human photographs, portraits, avatars and statues.

References in corpus (9)

• Conditional Generative Adversarial Nets
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)
• Perceptual Losses for Real-Time Style Transfer and Super-Resolution
• Autoencoding beyond pixels using a learned similarity metric
• Invertible Conditional GANs for image editing
• Learning Spatial Attention for Face Super-Resolution
• Toward Fine-grained Facial Expression Manipulation
• Masked Linear Regression for Learning Local Receptive Fields for Facial Expression Synthesis

Cited by in corpus (2)

• NeRF-AD: Neural Radiance Field with Attention-based Disentanglement for Talking Face Synthesis
• Improving Generative Adversarial Network Generalization for Facial Expression Synthesis