Deep Stereo using Adaptive Thin Volume Representation with Uncertainty Awareness

Abstract

We present Uncertainty-aware Cascaded Stereo Network (UCS-Net) for 3D reconstruction from multiple RGB images. Multi-view stereo (MVS) aims to reconstruct fine-grained scene geometry from multi-view images. Previous learning-based MVS methods estimate per-view depth using plane sweep volumes with a fixed depth hypothesis at each plane; this generally requires densely sampled planes for desired accuracy, and it is very hard to achieve high-resolution depth. In contrast, we propose adaptive thin volumes (ATVs); in an ATV, the depth hypothesis of each plane is spatially varying, which adapts to the uncertainties of previous per-pixel depth predictions. Our UCS-Net has three stages: the first stage processes a small standard plane sweep volume to predict low-resolution depth; two ATVs are then used in the following stages to refine the depth with higher resolution and higher accuracy. Our ATV consists of only a small number of planes; yet, it efficiently partitions local depth ranges within learned small intervals. In particular, we propose to use variance-based uncertainty estimates to adaptively construct ATVs; this differentiable process introduces reasonable and fine-grained spatial partitioning. Our multi-stage framework progressively subdivides the vast scene space with increasing depth resolution and precision, which enables scene reconstruction with high completeness and accuracy in a coarse-to-fine fashion. We demonstrate that our method achieves superior performance compared with state-of-the-art benchmarks on various challenging datasets.

Accepted to CVPR 2020 (Oral)

References in corpus (12)

• DeMoN: Depth and Motion Network for Learning Monocular Stereo
• Learning Representations and Generative Models for 3D Point Clouds
• Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue
• End-to-End Learning of Geometry and Context for Deep Stereo Regression
• Unsupervised Learning of Depth and Ego-Motion from Video
• Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction
• BA-Net: Dense Bundle Adjustment Network
• DeepStereo: Learning to Predict New Views from the World's Imagery
• DPSNet: End-to-end Deep Plane Sweep Stereo
• Learning to Reconstruct Shapes from Unseen Classes
• Shape Completion using 3D-Encoder-Predictor CNNs and Shape Synthesis
• Matryoshka Networks: Predicting 3D Geometry via Nested Shape Layers

Cited by in corpus (4)

• Neural Reflectance Fields for Appearance Acquisition
• SurfaceNet+: An End-to-end 3D Neural Network for Very Sparse Multi-view Stereopsis
• CFNet: Cascade and Fused Cost Volume for Robust Stereo Matching
• A Decomposition Model for Stereo Matching