Sparse-to-Dense: Depth Prediction from Sparse Depth Samples and a Single Image

Abstract

We consider the problem of dense depth prediction from a sparse set of depth measurements and a single RGB image. Since depth estimation from monocular images alone is inherently ambiguous and unreliable, to attain a higher level of robustness and accuracy, we introduce additional sparse depth samples, which are either acquired with a low-resolution depth sensor or computed via visual Simultaneous Localization and Mapping (SLAM) algorithms. We propose the use of a single deep regression network to learn directly from the RGB-D raw data, and explore the impact of number of depth samples on prediction accuracy. Our experiments show that, compared to using only RGB images, the addition of 100 spatially random depth samples reduces the prediction root-mean-square error by 50% on the NYU-Depth-v2 indoor dataset. It also boosts the percentage of reliable prediction from 59% to 92% on the KITTI dataset. We demonstrate two applications of the proposed algorithm: a plug-in module in SLAM to convert sparse maps to dense maps, and super-resolution for LiDARs. Software and video demonstration are publicly available.

accepted to ICRA 2018. 8 pages, 8 figures, 3 tables. Video at https://www.youtube.com/watch?v=vNIIT_M7x7Y. Code at https://github.com/fangchangma/sparse-to-dense

References in corpus (4)

• DepthTransfer: Depth Extraction from Video Using Non-parametric Sampling
• Depth Extraction from Video Using Non-parametric Sampling
• Unsupervised Learning of Depth and Ego-Motion from Video
• Semi-Supervised Deep Learning for Monocular Depth Map Prediction

Cited by in corpus (11)

• Confidence Propagation through CNNs for Guided Sparse Depth Regression
• Estimating Depth from RGB and Sparse Sensing
• Single Image Depth Estimation: An Overview
• Benchmarking Single Image Dehazing and Beyond
• Deep Robust Single Image Depth Estimation Neural Network Using Scene Understanding
• Deep Depth Completion of a Single RGB-D Image
• Revisiting Single Image Depth Estimation: Toward Higher Resolution Maps with Accurate Object Boundaries
• Advancing Self-supervised Monocular Depth Learning with Sparse LiDAR
• Plug-and-Play: Improve Depth Estimation via Sparse Data Propagation
• Deep Interpretable Non-Rigid Structure from Motion
• Balanced Depth Completion between Dense Depth Inference and Sparse Range Measurements via KISS-GP