Just-in-Time Reconstruction: Inpainting Sparse Maps using Single View Depth Predictors as Priors

Abstract

We present ``just-in-time reconstruction" as real-time image-guided inpainting of a map with arbitrary scale and sparsity to generate a fully dense depth map for the image. In particular, our goal is to inpaint a sparse map --- obtained from either a monocular visual SLAM system or a sparse sensor --- using a single-view depth prediction network as a virtual depth sensor. We adopt a fairly standard approach to data fusion, to produce a fused depth map by performing inference over a novel fully-connected Conditional Random Field (CRF) which is parameterized by the input depth maps and their pixel-wise confidence weights. Crucially, we obtain the confidence weights that parameterize the CRF model in a data-dependent manner via Convolutional Neural Networks (CNNs) which are trained to model the conditional depth error distributions given each source of input depth map and the associated RGB image. Our CRF model penalises absolute depth error in its nodes and pairwise scale-invariant depth error in its edges, and the confidence-based fusion minimizes the impact of outlier input depth values on the fused result. We demonstrate the flexibility of our method by real-time inpainting of ORB-SLAM, Kinect, and LIDAR depth maps acquired both indoors and outdoors at arbitrary scale and varied amount of irregular sparsity.

arXiv commentICRA 2018

References in corpus (7)

• ORB-SLAM: a Versatile and Accurate Monocular SLAM System
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields
• DeMoN: Depth and Motion Network for Learning Monocular Stereo

Cited by in corpus (9)

• Image inpainting: A review
• Sparse-to-Continuous: Enhancing Monocular Depth Estimation using Occupancy Maps
• See Through Smoke: Robust Indoor Mapping with Low-cost mmWave Radar
• FutureMapping 2: Gaussian Belief Propagation for Spatial AI
• Learned Semantic Multi-Sensor Depth Map Fusion
• EMPNet: Neural Localisation and Mapping Using Embedded Memory Points
• CAM-Convs: Camera-Aware Multi-Scale Convolutions for Single-View Depth
• Depth Completion via Deep Basis Fitting
• Environment reconstruction on depth images using Generative Adversarial Networks