SofGAN: A Portrait Image Generator with Dynamic Styling

Abstract

Recently, Generative Adversarial Networks (GANs)} have been widely used for portrait image generation. However, in the latent space learned by GANs, different attributes, such as pose, shape, and texture style, are generally entangled, making the explicit control of specific attributes difficult. To address this issue, we propose a SofGAN image generator to decouple the latent space of portraits into two subspaces: a geometry space and a texture space. The latent codes sampled from the two subspaces are fed to two network branches separately, one to generate the 3D geometry of portraits with canonical pose, and the other to generate textures. The aligned 3D geometries also come with semantic part segmentation, encoded as a semantic occupancy field (SOF). The SOF allows the rendering of consistent 2D semantic segmentation maps at arbitrary views, which are then fused with the generated texture maps and stylized to a portrait photo using our semantic instance-wise (SIW) module. Through extensive experiments, we show that our system can generate high quality portrait images with independently controllable geometry and texture attributes. The method also generalizes well in various applications such as appearance-consistent facial animation and dynamic styling.

Project page: https://apchenstu.github.io/sofgan/ Code: https://github.com/apchenstu/sofgan

References in corpus (6)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flows
• SEAN: Image Synthesis with Semantic Region-Adaptive Normalization
• First Order Motion Model for Image Animation
• BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images

Cited by in corpus (8)

• Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold
• StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis
• Facial-Sketch Synthesis: A New Challenge
• FreeStyleGAN: Free-view Editable Portrait Rendering with the Camera Manifold
• CGOF++: Controllable 3D Face Synthesis with Conditional Generative Occupancy Fields
• Do Inpainting Yourself: Generative Facial Inpainting Guided by Exemplars
• FACEMUG: A Multimodal Generative and Fusion Framework for Local Facial Editing
• DyStyle: Dynamic Neural Network for Multi-Attribute-Conditioned Style Editing