Light Field Reconstruction Using Convolutional Network on EPI and Extended Applications

Abstract

In this paper, a novel convolutional neural network (CNN)-based framework is developed for light field reconstruction from a sparse set of views. We indicate that the reconstruction can be efficiently modeled as angular restoration on an epipolar plane image (EPI). The main problem in direct reconstruction on the EPI involves an information asymmetry between the spatial and angular dimensions, where the detailed portion in the angular dimensions is damaged by undersampling. Directly upsampling or super-resolving the light field in the angular dimensions causes ghosting effects. To suppress these ghosting effects, we contribute a novel "blur-restoration-deblur" framework. First, the "blur" step is applied to extract the low-frequency components of the light field in the spatial dimensions by convolving each EPI slice with a selected blur kernel. Then, the "restoration" step is implemented by a CNN, which is trained to restore the angular details of the EPI. Finally, we use a non-blind "deblur" operation to recover the spatial high frequencies suppressed by the EPI blur. We evaluate our approach on several datasets, including synthetic scenes, real-world scenes and challenging microscope light field data. We demonstrate the high performance and robustness of the proposed framework compared with state-of-the-art algorithms. We further show extended applications, including depth enhancement and interpolation for unstructured input. More importantly, a novel rendering approach is presented by combining the proposed framework and depth information to handle large disparities.

Published in IEEE TPAMI, 2019

References in corpus (3)

• Caffe: Convolutional Architecture for Fast Feature Embedding
• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• Learning-Based View Synthesis for Light Field Cameras

Cited by in corpus (17)

• Disentangling Light Fields for Super-Resolution and Disparity Estimation
• Light Field Image Super-Resolution Using Deformable Convolution
• High-dimensional Dense Residual Convolutional Neural Network for Light Field Reconstruction
• Spatial-Angular Attention Network for Light Field Reconstruction
• Revisiting Light Field Rendering with Deep Anti-Aliasing Neural Network
• Fast Learning Radiance Fields by Shooting Much Fewer Rays
• Light Field View Synthesis via Aperture Disparity and Warping Confidence Map
• SurfaceNet+: An End-to-end 3D Neural Network for Very Sparse Multi-view Stereopsis
• Joint Spatial and Angular Super-Resolution from a Single Image
• 3DVSR: 3D EPI Volume-based Approach for Angular and Spatial Light field Image Super-resolution
• DeOccNet: Learning to See Through Foreground Occlusions in Light Fields
• High-Order Residual Network for Light Field Super-Resolution
• Deep Coarse-to-fine Dense Light Field Reconstruction with Flexible Sampling and Geometry-aware Fusion
• Light Field Saliency Detection with Deep Convolutional Networks
• Light Field Implicit Representation for Flexible Resolution Reconstruction
• Efficient Light Field Reconstruction via Spatio-Angular Dense Network
• Self-supervised Light Field View Synthesis Using Cycle Consistency