Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving

Abstract

3D object detection is an essential task in autonomous driving. Recent techniques excel with highly accurate detection rates, provided the 3D input data is obtained from precise but expensive LiDAR technology. Approaches based on cheaper monocular or stereo imagery data have, until now, resulted in drastically lower accuracies --- a gap that is commonly attributed to poor image-based depth estimation. However, in this paper we argue that it is not the quality of the data but its representation that accounts for the majority of the difference. Taking the inner workings of convolutional neural networks into consideration, we propose to convert image-based depth maps to pseudo-LiDAR representations --- essentially mimicking the LiDAR signal. With this representation we can apply different existing LiDAR-based detection algorithms. On the popular KITTI benchmark, our approach achieves impressive improvements over the existing state-of-the-art in image-based performance --- raising the detection accuracy of objects within the 30m range from the previous state-of-the-art of 22% to an unprecedented 74%. At the time of submission our algorithm holds the highest entry on the KITTI 3D object detection leaderboard for stereo-image-based approaches. Our code is publicly available at https://github.com/mileyan/pseudo_lidar.

arXiv commentAccepted by CVPR 2019

References in corpus (5)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• Deep Continuous Fusion for Multi-Sensor 3D Object Detection
• Depth Extraction from Video Using Non-parametric Sampling
• Anytime Stereo Image Depth Estimation on Mobile Devices

Cited by in corpus (12)

• Radar Voxel Fusion for 3D Object Detection
• SMOKE: Single-Stage Monocular 3D Object Detection via Keypoint Estimation
• MonoPair: Monocular 3D Object Detection Using Pairwise Spatial Relationships
• Attention-Based Depth Distillation with 3D-Aware Positional Encoding for Monocular 3D Object Detection
• Multi-Sensor 3D Object Box Refinement for Autonomous Driving
• YOLOStereo3D: A Step Back to 2D for Efficient Stereo 3D Detection
• Ground-aware Monocular 3D Object Detection for Autonomous Driving
• 3D Object Detection From LiDAR Data Using Distance Dependent Feature Extraction
• Monocular 3D Object Detection: An Extrinsic Parameter Free Approach
• Geometry-Aware Instance Segmentation with Disparity Maps
• Aug3D-RPN: Improving Monocular 3D Object Detection by Synthetic Images with Virtual Depth
• Joint Spatial-Temporal Optimization for Stereo 3D Object Tracking