Neural Rays for Occlusion-aware Image-based Rendering
arXiv:2107.13421
Abstract
We present a new neural representation, called Neural Ray (NeuRay), for the novel view synthesis task. Recent works construct radiance fields from image features of input views to render novel view images, which enables the generalization to new scenes. However, due to occlusions, a 3D point may be invisible to some input views. On such a 3D point, these generalization methods will include inconsistent image features from invisible views, which interfere with the radiance field construction. To solve this problem, we predict the visibility of 3D points to input views within our NeuRay representation. This visibility enables the radiance field construction to focus on visible image features, which significantly improves its rendering quality. Meanwhile, a novel consistency loss is proposed to refine the visibility in NeuRay when finetuning on a specific scene. Experiments demonstrate that our approach achieves state-of-the-art performance on the novel view synthesis task when generalizing to unseen scenes and outperforms per-scene optimization methods after finetuning.
CVPR2022. Project page https://liuyuan-pal.github.io/NeuRay/ Codes available at https://github.com/liuyuan-pal/NeuRay
References in corpus (11)
- NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
- Geo-PIFu: Geometry and Pixel Aligned Implicit Functions for Single-view Human Reconstruction
- NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction
- Neural Actor: Neural Free-view Synthesis of Human Actors with Pose Control
- ShaRF: Shape-conditioned Radiance Fields from a Single View
- MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo
- IBRNet: Learning Multi-View Image-Based Rendering
- PVA: Pixel-aligned Volumetric Avatars
- StereoPIFu: Depth Aware Clothed Human Digitization via Stereo Vision
- Fast Training of Neural Lumigraph Representations using Meta Learning
- Animatable Neural Radiance Fields for Modeling Dynamic Human Bodies