Learning Guided Convolutional Network for Depth Completion

Abstract

Dense depth perception is critical for autonomous driving and other robotics applications. However, modern LiDAR sensors only provide sparse depth measurement. It is thus necessary to complete the sparse LiDAR data, where a synchronized guidance RGB image is often used to facilitate this completion. Many neural networks have been designed for this task. However, they often na\"ıvely fuse the LiDAR data and RGB image information by performing feature concatenation or element-wise addition. Inspired by the guided image filtering, we design a novel guided network to predict kernel weights from the guidance image. These predicted kernels are then applied to extract the depth image features. In this way, our network generates content-dependent and spatially-variant kernels for multi-modal feature fusion. Dynamically generated spatially-variant kernels could lead to prohibitive GPU memory consumption and computation overhead. We further design a convolution factorization to reduce computation and memory consumption. The GPU memory reduction makes it possible for feature fusion to work in multi-stage scheme. We conduct comprehensive experiments to verify our method on real-world outdoor, indoor and synthetic datasets. Our method produces strong results. It outperforms state-of-the-art methods on the NYUv2 dataset and ranks 1st on the KITTI depth completion benchmark at the time of submission. It also presents strong generalization capability under different 3D point densities, various lighting and weather conditions as well as cross-dataset evaluations. The code will be released for reproduction.

arXiv commentSubmitted to the IEEE Transactions on Image Processing (TIP)

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Deep Bilateral Learning for Real-Time Image Enhancement
• Virtual Worlds as Proxy for Multi-Object Tracking Analysis
• Confidence Propagation through CNNs for Guided Sparse Depth Regression
• Context Encoding for Semantic Segmentation
• Learning Depth with Convolutional Spatial Propagation Network
• In Defense of Classical Image Processing: Fast Depth Completion on the CPU
• Deep Depth Completion of a Single RGB-D Image
• Sparse and Dense Data with CNNs: Depth Completion and Semantic Segmentation
• Dense Depth Posterior (DDP) from Single Image and Sparse Range

Cited by in corpus (9)

• Deep Learning for Image and Point Cloud Fusion in Autonomous Driving: A Review
• On Deep Learning Techniques to Boost Monocular Depth Estimation for Autonomous Navigation
• PENet: Towards Precise and Efficient Image Guided Depth Completion
• CSPN++: Learning Context and Resource Aware Convolutional Spatial Propagation Networks for Depth Completion
• Deep Learning based Monocular Depth Prediction: Datasets, Methods and Applications
• CodeVIO: Visual-Inertial Odometry with Learned Optimizable Dense Depth
• Depth Completion via Inductive Fusion of Planar LIDAR and Monocular Camera
• Self-supervised Visual-LiDAR Odometry with Flip Consistency
• Volumetric Propagation Network: Stereo-LiDAR Fusion for Long-Range Depth Estimation