Neural Radiance Flow for 4D View Synthesis and Video Processing
arXiv:2012.09790
Abstract
We present a method, Neural Radiance Flow (NeRFlow),to learn a 4D spatial-temporal representation of a dynamic scene from a set of RGB images. Key to our approach is the use of a neural implicit representation that learns to capture the 3D occupancy, radiance, and dynamics of the scene. By enforcing consistency across different modalities, our representation enables multi-view rendering in diverse dynamic scenes, including water pouring, robotic interaction, and real images, outperforming state-of-the-art methods for spatial-temporal view synthesis. Our approach works even when inputs images are captured with only one camera. We further demonstrate that the learned representation can serve as an implicit scene prior, enabling video processing tasks such as image super-resolution and de-noising without any additional supervision.
arXiv commentICCV 2021. Website: https://yilundu.github.io/nerflow/
References in corpus (9)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
- Embodied Hands: Modeling and Capturing Hands and Bodies Together
- Learning-Based View Synthesis for Light Field Cameras
- Generative and Discriminative Voxel Modeling with Convolutional Neural Networks
- Compressed Sensing with Deep Image Prior and Learned Regularization
- Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images
- Learning to Infer Implicit Surfaces without 3D Supervision
- Soft Rasterizer: Differentiable Rendering for Unsupervised Single-View Mesh Reconstruction