CIPS-3D: A 3D-Aware Generator of GANs Based on Conditionally-Independent Pixel Synthesis
arXiv:2110.09788
Abstract
The style-based GAN (StyleGAN) architecture achieved state-of-the-art results for generating high-quality images, but it lacks explicit and precise control over camera poses. The recently proposed NeRF-based GANs made great progress towards 3D-aware generators, but they are unable to generate high-quality images yet. This paper presents CIPS-3D, a style-based, 3D-aware generator that is composed of a shallow NeRF network and a deep implicit neural representation (INR) network. The generator synthesizes each pixel value independently without any spatial convolution or upsampling operation. In addition, we diagnose the problem of mirror symmetry that implies a suboptimal solution and solve it by introducing an auxiliary discriminator. Trained on raw, single-view images, CIPS-3D sets new records for 3D-aware image synthesis with an impressive FID of 6.97 for images at the resolution on FFHQ. We also demonstrate several interesting directions for CIPS-3D such as transfer learning and 3D-aware face stylization. The synthesis results are best viewed as videos, so we recommend the readers to check our github project at https://github.com/PeterouZh/CIPS-3D
arXiv comment3D-aware GANs based on NeRF, https://github.com/PeterouZh/CIPS-3D
References in corpus (20)
- Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- Large Scale GAN Training for High Fidelity Natural Image Synthesis
- Progressive Growing of GANs for Improved Quality, Stability, and Variation
- Alias-Free Generative Adversarial Networks
- Which Training Methods for GANs do actually Converge?
- NeRF++: Analyzing and Improving Neural Radiance Fields
- NeRF--: Neural Radiance Fields Without Known Camera Parameters
- PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization
- Closed-Form Factorization of Latent Semantics in GANs
- Resolution Dependent GAN Interpolation for Controllable Image Synthesis Between Domains
- Learning Implicit Fields for Generative Shape Modeling
- Visual Object Networks: Image Generation with Disentangled 3D Representation
- Semantic Hierarchy Emerges in Deep Generative Representations for Scene Synthesis
- UltraSR: Spatial Encoding is a Missing Key for Implicit Image Function-based Arbitrary-Scale Super-Resolution
- On the "steerability" of generative adversarial networks
- MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo
- Unsupervised Learning of Probably Symmetric Deformable 3D Objects from Images in the Wild
- Unsupervised Image-to-Image Translation via Pre-trained StyleGAN2 Network
- Local Deep Implicit Functions for 3D Shape