Estimating Depth from RGB and Sparse Sensing

Abstract

We present a deep model that can accurately produce dense depth maps given an RGB image with known depth at a very sparse set of pixels. The model works simultaneously for both indoor/outdoor scenes and produces state-of-the-art dense depth maps at nearly real-time speeds on both the NYUv2 and KITTI datasets. We surpass the state-of-the-art for monocular depth estimation even with depth values for only 1 out of every ~10000 image pixels, and we outperform other sparse-to-dense depth methods at all sparsity levels. With depth values for 1/256 of the image pixels, we achieve a mean absolute error of less than 1% of actual depth on indoor scenes, comparable to the performance of consumer-grade depth sensor hardware. Our experiments demonstrate that it would indeed be possible to efficiently transform sparse depth measurements obtained using e.g. lower-power depth sensors or SLAM systems into high-quality dense depth maps.

arXiv commentEuropean Conference on Computer Vision (ECCV) 2018. Updated to camera-ready version with additional experiments

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation
• Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics
• GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks
• An Overview of Depth Cameras and Range Scanners Based on Time-of-Flight Technologies
• Sparse-to-Dense: Depth Prediction from Sparse Depth Samples and a Single Image
• Single-Image Depth Perception in the Wild

Cited by in corpus (18)

• TransCG: A Large-Scale Real-World Dataset for Transparent Object Depth Completion and a Grasping Baseline
• A Survey of Simultaneous Localization and Mapping with an Envision in 6G Wireless Networks
• A Survey on Deep Learning Architectures for Image-based Depth Reconstruction
• Pixel-Accurate Depth Evaluation in Realistic Driving Scenarios
• Deep RGB-D Canonical Correlation Analysis For Sparse Depth Completion
• Adaptive LiDAR Sampling and Depth Completion using Ensemble Variance
• Plug-and-Play: Improve Depth Estimation via Sparse Data Propagation
• DeepPerimeter: Indoor Boundary Estimation from Posed Monocular Sequences
• Conf-Net: Toward High-Confidence Dense 3D Point-Cloud with Error-Map Prediction
• Deep Learning based Monocular Depth Prediction: Datasets, Methods and Applications
• Deep Interpretable Non-Rigid Structure from Motion
• Generating and Exploiting Probabilistic Monocular Depth Estimates
• FCFR-Net: Feature Fusion based Coarse-to-Fine Residual Learning for Depth Completion
• LiStereo: Generate Dense Depth Maps from LIDAR and Stereo Imagery
• Mesh Reconstruction from Aerial Images for Outdoor Terrain Mapping Using Joint 2D-3D Learning
• TerrainMesh: Metric-Semantic Terrain Reconstruction from Aerial Images Using Joint 2D-3D Learning
• High-frequency shape recovery from shading by CNN and domain adaptation
• Artificial Intelligence in Minimally Invasive Interventional Treatment