GANeRF: Leveraging Discriminators to Optimize Neural Radiance Fields
arXiv:2306.06044 · doi:10.1145/3618402
Abstract
Neural Radiance Fields (NeRF) have shown impressive novel view synthesis results; nonetheless, even thorough recordings yield imperfections in reconstructions, for instance due to poorly observed areas or minor lighting changes. Our goal is to mitigate these imperfections from various sources with a joint solution: we take advantage of the ability of generative adversarial networks (GANs) to produce realistic images and use them to enhance realism in 3D scene reconstruction with NeRFs. To this end, we learn the patch distribution of a scene using an adversarial discriminator, which provides feedback to the radiance field reconstruction, thus improving realism in a 3D-consistent fashion. Thereby, rendering artifacts are repaired directly in the underlying 3D representation by imposing multi-view path rendering constraints. In addition, we condition a generator with multi-resolution NeRF renderings which is adversarially trained to further improve rendering quality. We demonstrate that our approach significantly improves rendering quality, e.g., nearly halving LPIPS scores compared to Nerfacto while at the same time improving PSNR by 1.4dB on the advanced indoor scenes of Tanks and Temples.
arXiv commentSIGGRAPH Asia 2023, project page: https://barbararoessle.github.io/ganerf , video: https://youtu.be/352ccXWxQVE
References in corpus (18)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Instant Neural Graphics Primitives with a Multiresolution Hash Encoding
- LoRA: Low-Rank Adaptation of Large Language Models
- Which Training Methods for GANs do actually Converge?
- Nerfstudio: A Modular Framework for Neural Radiance Field Development
- NeRF++: Analyzing and Improving Neural Radiance Fields
- Perceptual Adversarial Networks for Image-to-Image Transformation
- Image inpainting: A review
- GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis
- Adding Conditional Control to Text-to-Image Diffusion Models
- MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction
- HumanRF: High-Fidelity Neural Radiance Fields for Humans in Motion
- HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields
- pixelNeRF: Neural Radiance Fields from One or Few Images
- EpiGRAF: Rethinking training of 3D GANs
- Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction
- 4K-NeRF: High Fidelity Neural Radiance Fields at Ultra High Resolutions