ApolloCar3D: A Large 3D Car Instance Understanding Benchmark for Autonomous Driving
arXiv:1811.12222
Abstract
Autonomous driving has attracted remarkable attention from both industry and academia. An important task is to estimate 3D properties(e.g.translation, rotation and shape) of a moving or parked vehicle on the road. This task, while critical, is still under-researched in the computer vision community - partially owing to the lack of large scale and fully-annotated 3D car database suitable for autonomous driving research. In this paper, we contribute the first large-scale database suitable for 3D car instance understanding - ApolloCar3D. The dataset contains 5,277 driving images and over 60K car instances, where each car is fitted with an industry-grade 3D CAD model with absolute model size and semantically labelled keypoints. This dataset is above 20 times larger than PASCAL3D+ and KITTI, the current state-of-the-art. To enable efficient labelling in 3D, we build a pipeline by considering 2D-3D keypoint correspondences for a single instance and 3D relationship among multiple instances. Equipped with such dataset, we build various baseline algorithms with the state-of-the-art deep convolutional neural networks. Specifically, we first segment each car with a pre-trained Mask R-CNN, and then regress towards its 3D pose and shape based on a deformable 3D car model with or without using semantic keypoints. We show that using keypoints significantly improves fitting performance. Finally, we develop a new 3D metric jointly considering 3D pose and 3D shape, allowing for comprehensive evaluation and ablation study. By comparing with human performance we suggest several future directions for further improvements.
References in corpus (15)
- Deep Convolutional Inverse Graphics Network
- Deep Continuous Fusion for Multi-Sensor 3D Object Detection
- Convolutional Pose Machines
- Deep Ordinal Regression Network for Monocular Depth Estimation
- Cascaded Pyramid Network for Multi-Person Pose Estimation
- Render for CNN: Viewpoint Estimation in Images Using CNNs Trained with Rendered 3D Model Views
- Towards Scene Understanding with Detailed 3D Object Representations
- Fully Convolutional Instance-aware Semantic Segmentation
- Deep Learning of Local RGB-D Patches for 3D Object Detection and 6D Pose Estimation
- Learning to Estimate 3D Human Pose and Shape from a Single Color Image
- Crafting a multi-task CNN for viewpoint estimation
- Deep Supervision with Shape Concepts for Occlusion-Aware 3D Object Parsing
- 3D Reconstruction of Simple Objects from A Single View Silhouette Image
- 3D Pose Estimation for Fine-Grained Object Categories
- Vehicle Pose and Shape Estimation through Multiple Monocular Vision