Learning to Synthesize a 4D RGBD Light Field from a Single Image
arXiv:1708.03292
Abstract
We present a machine learning algorithm that takes as input a 2D RGB image and synthesizes a 4D RGBD light field (color and depth of the scene in each ray direction). For training, we introduce the largest public light field dataset, consisting of over 3300 plenoptic camera light fields of scenes containing flowers and plants. Our synthesis pipeline consists of a convolutional neural network (CNN) that estimates scene geometry, a stage that renders a Lambertian light field using that geometry, and a second CNN that predicts occluded rays and non-Lambertian effects. Our algorithm builds on recent view synthesis methods, but is unique in predicting RGBD for each light field ray and improving unsupervised single image depth estimation by enforcing consistency of ray depths that should intersect the same scene point. Please see our supplementary video at https://youtu.be/yLCvWoQLnms
arXiv commentInternational Conference on Computer Vision (ICCV) 2017
References in corpus (11)
- Adam: A Method for Stochastic Optimization
- Deep Residual Learning for Image Recognition
- Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
- Multi-Scale Context Aggregation by Dilated Convolutions
- Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
- Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue
- Weakly-supervised Disentangling with Recurrent Transformations for 3D View Synthesis
- Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture
- Unsupervised Learning of 3D Structure from Images
- DeepStereo: Learning to Predict New Views from the World's Imagery
- A Point Set Generation Network for 3D Object Reconstruction from a Single Image