Pseudo-LiDAR++: Accurate Depth for 3D Object Detection in Autonomous Driving

Abstract

Detecting objects such as cars and pedestrians in 3D plays an indispensable role in autonomous driving. Existing approaches largely rely on expensive LiDAR sensors for accurate depth information. While recently pseudo-LiDAR has been introduced as a promising alternative, at a much lower cost based solely on stereo images, there is still a notable performance gap. In this paper we provide substantial advances to the pseudo-LiDAR framework through improvements in stereo depth estimation. Concretely, we adapt the stereo network architecture and loss function to be more aligned with accurate depth estimation of faraway objects --- currently the primary weakness of pseudo-LiDAR. Further, we explore the idea to leverage cheaper but extremely sparse LiDAR sensors, which alone provide insufficient information for 3D detection, to de-bias our depth estimation. We propose a depth-propagation algorithm, guided by the initial depth estimates, to diffuse these few exact measurements across the entire depth map. We show on the KITTI object detection benchmark that our combined approach yields substantial improvements in depth estimation and stereo-based 3D object detection --- outperforming the previous state-of-the-art detection accuracy for faraway objects by 40%. Our code is available at https://github.com/mileyan/Pseudo_Lidar_V2.

arXiv commentAccepted to International Conference on Learning Representations (ICLR) 2020

References in corpus (6)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Deep Continuous Fusion for Multi-Sensor 3D Object Detection
• HDNET: Exploiting HD Maps for 3D Object Detection
• A General Pipeline for 3D Detection of Vehicles
• Plug-and-Play: Improve Depth Estimation via Sparse Data Propagation
• Dense Depth Posterior (DDP) from Single Image and Sparse Range

Cited by in corpus (29)

• Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges
• Cross-Modality Knowledge Distillation Network for Monocular 3D Object Detection
• Radar Voxel Fusion for 3D Object Detection
• Progressive Coordinate Transforms for Monocular 3D Object Detection
• Harnessing Interpretable and Unsupervised Machine Learning to Address Big Data from Modern X-ray Diffraction
• CLOCs: Camera-LiDAR Object Candidates Fusion for 3D Object Detection
• BEV-Seg: Bird's Eye View Semantic Segmentation Using Geometry and Semantic Point Cloud
• Autonomous Driving with Deep Learning: A Survey of State-of-Art Technologies
• PV-RCNN++: Point-Voxel Feature Set Abstraction With Local Vector Representation for 3D Object Detection
• DSGN: Deep Stereo Geometry Network for 3D Object Detection
• Advancing Self-supervised Monocular Depth Learning with Sparse LiDAR
• Depth-conditioned Dynamic Message Propagation for Monocular 3D Object Detection
• Single-Shot 3D Detection of Vehicles from Monocular RGB Images via Geometry Constrained Keypoints in Real-Time
• SelfVoxeLO: Self-supervised LiDAR Odometry with Voxel-based Deep Neural Networks
• Rethinking Pseudo-LiDAR Representation
• ZoomNet: Part-Aware Adaptive Zooming Neural Network for 3D Object Detection
• CamLiFlow: Bidirectional Camera-LiDAR Fusion for Joint Optical Flow and Scene Flow Estimation
• GAN-Based LiDAR Intensity Simulation
• Adjusting Bias in Long Range Stereo Matching: A semantics guided approach
• FusionMapping: Learning Depth Prediction with Monocular Images and 2D Laser Scans
• Aug3D-RPN: Improving Monocular 3D Object Detection by Synthetic Images with Virtual Depth
• Lite-FPN for Keypoint-based Monocular 3D Object Detection
• ISETAuto: Detecting vehicles with depth and radiance information
• Camera Pose Matters: Improving Depth Prediction by Mitigating Pose Distribution Bias
• CubifAE-3D: Monocular Camera Space Cubification for Auto-Encoder based 3D Object Detection
• Expanding Sparse Guidance for Stereo Matching
• Scalable Primitives for Generalized Sensor Fusion in Autonomous Vehicles
• AutoLay: Benchmarking amodal layout estimation for autonomous driving
• Achieving Domain Robustness in Stereo Matching Networks by Removing Shortcut Learning