IBRNet: Learning Multi-View Image-Based Rendering
arXiv:2102.13090
Abstract
We present a method that synthesizes novel views of complex scenes by interpolating a sparse set of nearby views. The core of our method is a network architecture that includes a multilayer perceptron and a ray transformer that estimates radiance and volume density at continuous 5D locations (3D spatial locations and 2D viewing directions), drawing appearance information on the fly from multiple source views. By drawing on source views at render time, our method hearkens back to classic work on image-based rendering (IBR), and allows us to render high-resolution imagery. Unlike neural scene representation work that optimizes per-scene functions for rendering, we learn a generic view interpolation function that generalizes to novel scenes. We render images using classic volume rendering, which is fully differentiable and allows us to train using only multi-view posed images as supervision. Experiments show that our method outperforms recent novel view synthesis methods that also seek to generalize to novel scenes. Further, if fine-tuned on each scene, our method is competitive with state-of-the-art single-scene neural rendering methods. Project page: https://ibrnet.github.io/
CVPR 2021. Project page: https://ibrnet.github.io/
References in corpus (5)
- Instance Normalization: The Missing Ingredient for Fast Stylization
- The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
- Learning-Based View Synthesis for Light Field Cameras
- PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization
- Adversarial Texture Optimization from RGB-D Scans
Cited by in corpus (7)
- Neural Human Performer: Learning Generalizable Radiance Fields for Human Performance Rendering
- ShaRF: Shape-conditioned Radiance Fields from a Single View
- NeRF in detail: Learning to sample for view synthesis
- Single-image Full-body Human Relighting
- Fast Training of Neural Lumigraph Representations using Meta Learning
- Neural Rays for Occlusion-aware Image-based Rendering
- iButter: Neural Interactive Bullet Time Generator for Human Free-viewpoint Rendering