Radar-Camera Pixel Depth Association for Depth Completion

Abstract

While radar and video data can be readily fused at the detection level, fusing them at the pixel level is potentially more beneficial. This is also more challenging in part due to the sparsity of radar, but also because automotive radar beams are much wider than a typical pixel combined with a large baseline between camera and radar, which results in poor association between radar pixels and color pixel. A consequence is that depth completion methods designed for LiDAR and video fare poorly for radar and video. Here we propose a radar-to-pixel association stage which learns a mapping from radar returns to pixels. This mapping also serves to densify radar returns. Using this as a first stage, followed by a more traditional depth completion method, we are able to achieve image-guided depth completion with radar and video. We demonstrate performance superior to camera and radar alone on the nuScenes dataset. Our source code is available at https://github.com/longyunf/rc-pda.

arXiv commentIEEE Conference on Computer Vision and Pattern Recognition, 2021

References in corpus (8)

• Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges
• From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimation
• SfM-Net: Learning of Structure and Motion from Video
• Scalability in Perception for Autonomous Driving: Waymo Open Dataset
• Argoverse: 3D Tracking and Forecasting with Rich Maps
• Seeing Around Street Corners: Non-Line-of-Sight Detection and Tracking In-the-Wild Using Doppler Radar
• Unsupervised Learning of Geometry with Edge-aware Depth-Normal Consistency
• ConnNet: A Long-Range Relation-Aware Pixel-Connectivity Network for Salient Segmentation