Learning a Multi-View Stereo Machine

Abstract

We present a learnt system for multi-view stereopsis. In contrast to recent learning based methods for 3D reconstruction, we leverage the underlying 3D geometry of the problem through feature projection and unprojection along viewing rays. By formulating these operations in a differentiable manner, we are able to learn the system end-to-end for the task of metric 3D reconstruction. End-to-end learning allows us to jointly reason about shape priors while conforming geometric constraints, enabling reconstruction from much fewer images (even a single image) than required by classical approaches as well as completion of unseen surfaces. We thoroughly evaluate our approach on the ShapeNet dataset and demonstrate the benefits over classical approaches as well as recent learning based methods.

References in corpus (6)

• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue
• End-to-End Learning of Geometry and Context for Deep Stereo Regression
• Unsupervised Learning of Depth and Ego-Motion from Video
• Unsupervised Learning of 3D Structure from Images
• OctNetFusion: Learning Depth Fusion from Data

Cited by in corpus (16)

• The ApolloScape Open Dataset for Autonomous Driving and its Application
• A Survey on Deep Learning Techniques for Stereo-based Depth Estimation
• Dense 3D Object Reconstruction from a Single Depth View
• GIFS: Neural Implicit Function for General Shape Representation
• DeepSketchHair: Deep Sketch-based 3D Hair Modeling
• SurfaceNet+: An End-to-end 3D Neural Network for Very Sparse Multi-view Stereopsis
• DSGN: Deep Stereo Geometry Network for 3D Object Detection
• GARNet: Global-Aware Multi-View 3D Reconstruction Network and the Cost-Performance Tradeoff
• Remote Sensing Novel View Synthesis with Implicit Multiplane Representations
• NeeDrop: Self-supervised Shape Representation from Sparse Point Clouds using Needle Dropping
• Learning Pose-invariant 3D Object Reconstruction from Single-view Images
• Weakly Supervised Learning of Multi-Object 3D Scene Decompositions Using Deep Shape Priors
• Semantic Photometric Bundle Adjustment on Natural Sequences
• Learning a Hierarchical Latent-Variable Model of 3D Shapes
• Learning Single-Image Depth from Videos using Quality Assessment Networks
• Visual Enhanced 3D Point Cloud Reconstruction from A Single Image