The ApolloScape Open Dataset for Autonomous Driving and its Application

Abstract

Autonomous driving has attracted tremendous attention especially in the past few years. The key techniques for a self-driving car include solving tasks like 3D map construction, self-localization, parsing the driving road and understanding objects, which enable vehicles to reason and act. However, large scale data set for training and system evaluation is still a bottleneck for developing robust perception models. In this paper, we present the ApolloScape dataset [1] and its applications for autonomous driving. Compared with existing public datasets from real scenes, e.g. KITTI [2] or Cityscapes [3], ApolloScape contains much large and richer labelling including holistic semantic dense point cloud for each site, stereo, per-pixel semantic labelling, lanemark labelling, instance segmentation, 3D car instance, high accurate location for every frame in various driving videos from multiple sites, cities and daytimes. For each task, it contains at lease 15x larger amount of images than SOTA datasets. To label such a complete dataset, we develop various tools and algorithms specified for each task to accelerate the labelling process, such as 3D-2D segment labeling tools, active labelling in videos etc. Depend on ApolloScape, we are able to develop algorithms jointly consider the learning and inference of multiple tasks. In this paper, we provide a sensor fusion scheme integrating camera videos, consumer-grade motion sensors (GPS/IMU), and a 3D semantic map in order to achieve robust self-localization and semantic segmentation for autonomous driving. We show that practically, sensor fusion and joint learning of multiple tasks are beneficial to achieve a more robust and accurate system. We expect our dataset and proposed relevant algorithms can support and motivate researchers for further development of multi-sensor fusion and multi-task learning in the field of computer vision.

Version 4: Accepted by TPAMI. Version 3: 17 pages, 10 tables, 11 figures, added the application (DeLS-3D) based on the ApolloScape Dataset. Version 2: 7 pages, 6 figures, added comparison with BDD100K dataset

References in corpus (11)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Rethinking Atrous Convolution for Semantic Image Segmentation
• ORB-SLAM: a Versatile and Accurate Monocular SLAM System
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation
• DeMoN: Depth and Motion Network for Learning Monocular Stereo
• FCNs in the Wild: Pixel-level Adversarial and Constraint-based Adaptation
• Curriculum Domain Adaptation for Semantic Segmentation of Urban Scenes
• Wider or Deeper: Revisiting the ResNet Model for Visual Recognition
• Learning a Multi-View Stereo Machine

Cited by in corpus (69)

• A Survey of Autonomous Driving: Common Practices and Emerging Technologies
• Perception and Sensing for Autonomous Vehicles Under Adverse Weather Conditions: A Survey
• Milestones in Autonomous Driving and Intelligent Vehicles: Survey of Surveys
• A Survey on Deep Learning Techniques for Stereo-based Depth Estimation
• A Review and Comparative Study on Probabilistic Object Detection in Autonomous Driving
• A2D2: Audi Autonomous Driving Dataset
• Radar-Camera Fusion for Object Detection and Semantic Segmentation in Autonomous Driving: A Comprehensive Review
• OverlapTransformer: An Efficient and Rotation-Invariant Transformer Network for LiDAR-Based Place Recognition
• One Thousand and One Hours: Self-driving Motion Prediction Dataset
• Augmented LiDAR Simulator for Autonomous Driving
• Pedestrian Models for Autonomous Driving Part II: High-Level Models of Human Behavior
• Reference Pose Generation for Long-term Visual Localization via Learned Features and View Synthesis
• JRDB: A Dataset and Benchmark of Egocentric Robot Visual Perception of Humans in Built Environments
• A Survey on Approximate Edge AI for Energy Efficient Autonomous Driving Services
• A Survey on Deep Learning for Localization and Mapping: Towards the Age of Spatial Machine Intelligence
• ROAD: The ROad event Awareness Dataset for Autonomous Driving
• A Comprehensive Analysis of Weakly-Supervised Semantic Segmentation in Different Image Domains
• TJU-DHD: A Diverse High-Resolution Dataset for Object Detection
• GRIP++: Enhanced Graph-based Interaction-aware Trajectory Prediction for Autonomous Driving
• Deep Learning-based Depth Estimation Methods from Monocular Image and Videos: A Comprehensive Survey
• FPCC: Fast Point Cloud Clustering based Instance Segmentation for Industrial Bin-picking
• A Survey on RGB-D Datasets
• Brno Urban Dataset -- The New Data for Self-Driving Agents and Mapping Tasks
• A Comprehensive Review of Modern Object Segmentation Approaches
• Heterogeneous Trajectory Forecasting via Risk and Scene Graph Learning
• Overview of Tools Supporting Planning for Automated Driving
• Optimal and Robust Category-level Perception: Object Pose and Shape Estimation from 2D and 3D Semantic Keypoints
• Optimal Pose and Shape Estimation for Category-level 3D Object Perception
• Ad-datasets: a meta-collection of data sets for autonomous driving
• Secure Deep Learning Engineering: A Software Quality Assurance Perspective
• Multi-Modal 3D Object Detection in Autonomous Driving: a Survey
• MUSES: The Multi-Sensor Semantic Perception Dataset for Driving under Uncertainty
• Video Unsupervised Domain Adaptation with Deep Learning: A Comprehensive Survey
• BAAI-VANJEE Roadside Dataset: Towards the Connected Automated Vehicle Highway technologies in Challenging Environments of China
• MODISSA: a multipurpose platform for the prototypical realization of vehicle-related applications using optical sensors
• EU Long-term Dataset with Multiple Sensors for Autonomous Driving
• Multi-LiCa: A Motion and Targetless Multi LiDAR-to-LiDAR Calibration Framework
• Boosting Real-Time Driving Scene Parsing with Shared Semantics
• CARRADA Dataset: Camera and Automotive Radar with Range-Angle-Doppler Annotations
• Inter-Region Affinity Distillation for Road Marking Segmentation
• Channel Attention based Iterative Residual Learning for Depth Map Super-Resolution
• An Automotive Case Study on the Limits of Approximation for Object Detection
• Interaction-Aware Trajectory Prediction of Connected Vehicles using CNN-LSTM Networks
• STRIDE: Street View-based Environmental Feature Detection and Pedestrian Collision Prediction
• Learning Depth from Monocular Videos Using Synthetic Data: A Temporally-Consistent Domain Adaptation Approach
• Navigation Agents for the Visually Impaired: A Sidewalk Simulator and Experiments
• SynWoodScape: Synthetic Surround-view Fisheye Camera Dataset for Autonomous Driving
• SS3D: Single Shot 3D Object Detector
• An evaluation of Deep Learning based stereo dense matching dataset shift from aerial images and a large scale stereo dataset
• Are We Hungry for 3D LiDAR Data for Semantic Segmentation? A Survey and Experimental Study
• Towards Scenario- and Capability-Driven Dataset Development and Evaluation: An Approach in the Context of Mapless Automated Driving
• Vehicle Telematics Via Exteroceptive Sensors: A Survey
• DSOR: A Scalable Statistical Filter for Removing Falling Snow from LiDAR Point Clouds in Severe Winter Weather
• Transfer beyond the Field of View: Dense Panoramic Semantic Segmentation via Unsupervised Domain Adaptation
• RaidaR: A Rich Annotated Image Dataset of Rainy Street Scenes
• Vehicle Reconstruction and Texture Estimation Using Deep Implicit Semantic Template Mapping
• AutoLay: Benchmarking amodal layout estimation for autonomous driving
• Interactive Annotation of 3D Object Geometry using 2D Scribbles
• Tackling Variabilities in Autonomous Driving
• End-to-End Monocular Vanishing Point Detection Exploiting Lane Annotations
• Feasibility of Video-based Sub-meter Localization on Resource-constrained Platforms
• IAFA: Instance-aware Feature Aggregation for 3D Object Detection from a Single Image
• Brno Urban Dataset: Winter Extention
• On the Synergies between Machine Learning and Binocular Stereo for Depth Estimation from Images: a Survey
• CVPR 2019 WAD Challenge on Trajectory Prediction and 3D Perception
• Predicting Depth from Semantic Segmentation using Game Engine Dataset
• CAR -- Cityscapes Attributes Recognition A Multi-category Attributes Dataset for Autonomous Vehicles
• PlantStereo: A Stereo Matching Benchmark for Plant Surface Dense Reconstruction
• A Data Efficiency Study of Synthetic Fog for Object Detection Using the Clear2Fog Pipeline