FreeStyleGAN: Free-view Editable Portrait Rendering with the Camera Manifold

Abstract

Current Generative Adversarial Networks (GANs) produce photorealistic renderings of portrait images. Embedding real images into the latent space of such models enables high-level image editing. While recent methods provide considerable semantic control over the (re-)generated images, they can only generate a limited set of viewpoints and cannot explicitly control the camera. Such 3D camera control is required for 3D virtual and mixed reality applications. In our solution, we use a few images of a face to perform 3D reconstruction, and we introduce the notion of the GAN camera manifold, the key element allowing us to precisely define the range of images that the GAN can reproduce in a stable manner. We train a small face-specific neural implicit representation network to map a captured face to this manifold and complement it with a warping scheme to obtain free-viewpoint novel-view synthesis. We show how our approach - due to its precise camera control - enables the integration of a pre-trained StyleGAN into standard 3D rendering pipelines, allowing e.g., stereo rendering or consistent insertion of faces in synthetic 3D environments. Our solution proposes the first truly free-viewpoint rendering of realistic faces at interactive rates, using only a small number of casual photos as input, while simultaneously allowing semantic editing capabilities, such as facial expression or lighting changes.

arXiv commentProject webpage: https://repo-sam.inria.fr/fungraph/freestylegan/

References in corpus (18)

• Adam: A Method for Stochastic Optimization
• VoxCeleb: a large-scale speaker identification dataset
• How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)
• Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
• The Perception-Distortion Tradeoff
• Training Generative Adversarial Networks with Limited Data
• Alias-Free Generative Adversarial Networks
• Neural Volumes: Learning Dynamic Renderable Volumes from Images
• NeRF++: Analyzing and Improving Neural Radiance Fields
• StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flows
• GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis
• Deep Appearance Models for Face Rendering
• Editable Free-viewpoint Video Using a Layered Neural Representation
• SofGAN: A Portrait Image Generator with Dynamic Styling
• Portrait Neural Radiance Fields from a Single Image
• E-LPIPS: Robust Perceptual Image Similarity via Random Transformation Ensembles
• Improving Inversion and Generation Diversity in StyleGAN using a Gaussianized Latent Space
• PhotoApp: Photorealistic Appearance Editing of Head Portraits

Cited by in corpus (2)

• Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold
• CGOF++: Controllable 3D Face Synthesis with Conditional Generative Occupancy Fields