Stereo Magnification: Learning View Synthesis using Multiplane Images
arXiv:1805.09817
Abstract
The view synthesis problem--generating novel views of a scene from known imagery--has garnered recent attention due in part to compelling applications in virtual and augmented reality. In this paper, we explore an intriguing scenario for view synthesis: extrapolating views from imagery captured by narrow-baseline stereo cameras, including VR cameras and now-widespread dual-lens camera phones. We call this problem stereo magnification, and propose a learning framework that leverages a new layered representation that we call multiplane images (MPIs). Our method also uses a massive new data source for learning view extrapolation: online videos on YouTube. Using data mined from such videos, we train a deep network that predicts an MPI from an input stereo image pair. This inferred MPI can then be used to synthesize a range of novel views of the scene, including views that extrapolate significantly beyond the input baseline. We show that our method compares favorably with several recent view synthesis methods, and demonstrate applications in magnifying narrow-baseline stereo images.
arXiv commentAccepted to SIGGRAPH 2018. Project webpage: https://people.eecs.berkeley.edu/~tinghuiz/projects/mpi/
References in corpus (1)
Cited by in corpus (27)
- Neural Volumes: Learning Dynamic Renderable Volumes from Images
- DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks
- NeRF-SR: High-Quality Neural Radiance Fields using Supersampling
- Revisiting Light Field Rendering with Deep Anti-Aliasing Neural Network
- Neural Human Performer: Learning Generalizable Radiance Fields for Human Performance Rendering
- Physics Informed Neural Fields for Smoke Reconstruction with Sparse Data
- ViP-NeRF: Visibility Prior for Sparse Input Neural Radiance Fields
- DoF-NeRF: Depth-of-Field Meets Neural Radiance Fields
- MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo
- SimpleNeRF: Regularizing Sparse Input Neural Radiance Fields with Simpler Solutions
- Remote Sensing Novel View Synthesis with Implicit Multiplane Representations
- Acquiring a Dynamic Light Field through a Single-Shot Coded Image
- IE-NeRF: Inpainting Enhanced Neural Radiance Fields in the Wild
- VET: Visual Error Tomography for Point Cloud Completion and High-Quality Neural Rendering
- Robust Semi-Supervised Monocular Depth Estimation with Reprojected Distances
- Breaking the Spatio-Angular Trade-off for Light Field Super-Resolution via LSTM Modelling on Epipolar Plane Images
- Web Stereo Video Supervision for Depth Prediction from Dynamic Scenes
- Learning Object-Compositional Neural Radiance Field for Editable Scene Rendering
- KinePose: A temporally optimized inverse kinematics technique for 6DOF human pose estimation with biomechanical constraints
- A Neural Rendering Framework for Free-Viewpoint Relighting
- Reconstructing Continuous Light Field From Single Coded Image
- Multi-View Matching (MVM): Facilitating Multi-Person 3D Pose Estimation Learning with Action-Frozen People Video
- Aug3D-RPN: Improving Monocular 3D Object Detection by Synthetic Images with Virtual Depth
- Learning Indoor Inverse Rendering with 3D Spatially-Varying Lighting
- NoPe-NeRF++: Local-to-Global Optimization of NeRF with No Pose Prior
- Distortion Reduction for Off-Center Perspective Projection of Panoramas
- ADeLA: Automatic Dense Labeling with Attention for Viewpoint Adaptation in Semantic Segmentation