CGOF++: Controllable 3D Face Synthesis with Conditional Generative Occupancy Fields

Abstract

Capitalizing on the recent advances in image generation models, existing controllable face image synthesis methods are able to generate high-fidelity images with some levels of controllability, e.g., controlling the shapes, expressions, textures, and poses of the generated face images. However, previous methods focus on controllable 2D image generative models, which are prone to producing inconsistent face images under large expression and pose changes. In this paper, we propose a new NeRF-based conditional 3D face synthesis framework, which enables 3D controllability over the generated face images by imposing explicit 3D conditions from 3D face priors. At its core is a conditional Generative Occupancy Field (cGOF++) that effectively enforces the shape of the generated face to conform to a given 3D Morphable Model (3DMM) mesh, built on top of EG3D [1], a recent tri-plane-based generative model. To achieve accurate control over fine-grained 3D face shapes of the synthesized images, we additionally incorporate a 3D landmark loss as well as a volume warping loss into our synthesis framework. Experiments validate the effectiveness of the proposed method, which is able to generate high-fidelity face images and shows more precise 3D controllability than state-of-the-art 2D-based controllable face synthesis methods.

Accepted to IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI). This article is an extension of the NeurIPS'22 paper arXiv:2206.08361

References in corpus (12)

• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• Alias-Free Generative Adversarial Networks
• StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis
• Unsupervised Generative 3D Shape Learning from Natural Images
• Generative Neural Articulated Radiance Fields
• Inverse Graphics GAN: Learning to Generate 3D Shapes from Unstructured 2D Data
• Face Synthesis from Visual Attributes via Sketch using Conditional VAEs and GANs
• Sem2NeRF: Converting Single-View Semantic Masks to Neural Radiance Fields
• AniFaceGAN: Animatable 3D-Aware Face Image Generation for Video Avatars
• FreeStyleGAN: Free-view Editable Portrait Rendering with the Camera Manifold
• Controllable 3D Face Synthesis with Conditional Generative Occupancy Fields
• CGOF++: Controllable 3D Face Synthesis with Conditional Generative Occupancy Fields

Cited by in corpus (2)

• SceneDreamer: Unbounded 3D Scene Generation from 2D Image Collections
• CGOF++: Controllable 3D Face Synthesis with Conditional Generative Occupancy Fields