GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis

Abstract

While 2D generative adversarial networks have enabled high-resolution image synthesis, they largely lack an understanding of the 3D world and the image formation process. Thus, they do not provide precise control over camera viewpoint or object pose. To address this problem, several recent approaches leverage intermediate voxel-based representations in combination with differentiable rendering. However, existing methods either produce low image resolution or fall short in disentangling camera and scene properties, e.g., the object identity may vary with the viewpoint. In this paper, we propose a generative model for radiance fields which have recently proven successful for novel view synthesis of a single scene. In contrast to voxel-based representations, radiance fields are not confined to a coarse discretization of the 3D space, yet allow for disentangling camera and scene properties while degrading gracefully in the presence of reconstruction ambiguity. By introducing a multi-scale patch-based discriminator, we demonstrate synthesis of high-resolution images while training our model from unposed 2D images alone. We systematically analyze our approach on several challenging synthetic and real-world datasets. Our experiments reveal that radiance fields are a powerful representation for generative image synthesis, leading to 3D consistent models that render with high fidelity.

References in corpus (17)

• Auto-Encoding Variational Bayes
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Spectral Normalization for Generative Adversarial Networks
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• Which Training Methods for GANs do actually Converge?
• Learning Representations and Generative Models for 3D Point Clouds
• Generative and Discriminative Voxel Modeling with Convolutional Neural Networks
• On the Spectral Bias of Neural Networks
• PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization
• BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images
• PhotoShape: Photorealistic Materials for Large-Scale Shape Collections
• Visual Object Networks: Image Generation with Disentangled 3D Representation
• Detecting GAN generated Fake Images using Co-occurrence Matrices
• Source Generator Attribution via Inversion

Cited by in corpus (20)

• DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks
• Reconstructing Personalized Semantic Facial NeRF Models From Monocular Video
• Locally Attentional SDF Diffusion for Controllable 3D Shape Generation
• A survey of synthetic data augmentation methods in computer vision
• Physics Informed Neural Fields for Smoke Reconstruction with Sparse Data
• Generative Adversarial Networks for Image and Video Synthesis: Algorithms and Applications
• DoF-NeRF: Depth-of-Field Meets Neural Radiance Fields
• Mip-NeRF RGB-D: Depth Assisted Fast Neural Radiance Fields
• GANeRF: Leveraging Discriminators to Optimize Neural Radiance Fields
• FreeStyleGAN: Free-view Editable Portrait Rendering with the Camera Manifold
• Benchmarking Neural Radiance Fields for Autonomous Robots: An Overview
• Blended-NeRF: Zero-Shot Object Generation and Blending in Existing Neural Radiance Fields
• CGOF++: Controllable 3D Face Synthesis with Conditional Generative Occupancy Fields
• Designing An Illumination-Aware Network for Deep Image Relighting
• Deep Deformable 3D Caricatures with Learned Shape Control
• MotionVideoGAN: A Novel Video Generator Based on the Motion Space Learned from Image Pairs
• Human Image Generation: A Comprehensive Survey
• SlimmeRF: Slimmable Radiance Fields
• Make Your Brief Stroke Real and Stereoscopic: 3D-Aware Simplified Sketch to Portrait Generation
• SAT3D: Image-driven Semantic Attribute Transfer in 3D