StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis

Abstract

We propose StyleNeRF, a 3D-aware generative model for photo-realistic high-resolution image synthesis with high multi-view consistency, which can be trained on unstructured 2D images. Existing approaches either cannot synthesize high-resolution images with fine details or yield noticeable 3D-inconsistent artifacts. In addition, many of them lack control over style attributes and explicit 3D camera poses. StyleNeRF integrates the neural radiance field (NeRF) into a style-based generator to tackle the aforementioned challenges, i.e., improving rendering efficiency and 3D consistency for high-resolution image generation. We perform volume rendering only to produce a low-resolution feature map and progressively apply upsampling in 2D to address the first issue. To mitigate the inconsistencies caused by 2D upsampling, we propose multiple designs, including a better upsampler and a new regularization loss. With these designs, StyleNeRF can synthesize high-resolution images at interactive rates while preserving 3D consistency at high quality. StyleNeRF also enables control of camera poses and different levels of styles, which can generalize to unseen views. It also supports challenging tasks, including zoom-in and-out, style mixing, inversion, and semantic editing.

arXiv comment24 pages, 19 figures. Project page: http://jiataogu.me/style_nerf/

References in corpus (14)

• Learning Transferable Visual Models From Natural Language Supervision
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Self-Attention Generative Adversarial Networks
• Alias-Free Generative Adversarial Networks
• Which Training Methods for GANs do actually Converge?
• NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
• NeRF++: Analyzing and Improving Neural Radiance Fields
• Making Convolutional Networks Shift-Invariant Again
• The Deepfake Detection Challenge (DFDC) Preview Dataset
• SofGAN: A Portrait Image Generator with Dynamic Styling
• Dropout-GAN: Learning from a Dynamic Ensemble of Discriminators
• StyleRig: Rigging StyleGAN for 3D Control over Portrait Images
• On Noise Injection in Generative Adversarial Networks

Cited by in corpus (11)

• Face Generation and Editing with StyleGAN: A Survey
• Flexible Isosurface Extraction for Gradient-Based Mesh Optimization
• Physics Informed Neural Fields for Smoke Reconstruction with Sparse Data
• Blended-NeRF: Zero-Shot Object Generation and Blending in Existing Neural Radiance Fields
• CGOF++: Controllable 3D Face Synthesis with Conditional Generative Occupancy Fields
• MotionVideoGAN: A Novel Video Generator Based on the Motion Space Learned from Image Pairs
• Placing Human Animations into 3D Scenes by Learning Interaction- and Geometry-Driven Keyframes
• Multi-NeuS: 3D Head Portraits from Single Image with Neural Implicit Functions
• SlimmeRF: Slimmable Radiance Fields
• Human Image Generation: A Comprehensive Survey
• PACE: Data-Driven Virtual Agent Interaction in Dense and Cluttered Environments