TorontoCity: Seeing the World with a Million Eyes

Abstract

In this paper we introduce the TorontoCity benchmark, which covers the full greater Toronto area (GTA) with 712.5 $k{m}^2$ of land, 8439 $km$ of road and around 400,000 buildings. Our benchmark provides different perspectives of the world captured from airplanes, drones and cars driving around the city. Manually labeling such a large scale dataset is infeasible. Instead, we propose to utilize different sources of high-precision maps to create our ground truth. Towards this goal, we develop algorithms that allow us to align all data sources with the maps while requiring minimal human supervision. We have designed a wide variety of tasks including building height estimation (reconstruction), road centerline and curb extraction, building instance segmentation, building contour extraction (reorganization), semantic labeling and scene type classification (recognition). Our pilot study shows that most of these tasks are still difficult for modern convolutional neural networks.

References in corpus (8)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Going Deeper with Convolutions
• The Cityscapes Dataset for Semantic Urban Scene Understanding
• Fully Convolutional Networks for Semantic Segmentation
• DeepDriving: Learning Affordance for Direct Perception in Autonomous Driving
• Playing for Data: Ground Truth from Computer Games
• SceneNet: Understanding Real World Indoor Scenes With Synthetic Data
• Generic 3D Representation via Pose Estimation and Matching

Cited by in corpus (10)

• SpaceNet: A Remote Sensing Dataset and Challenge Series
• City-Scale Road Extraction from Satellite Imagery v2: Road Speeds and Travel Times
• Learning deep structured active contours end-to-end
• An Entropic Optimal Transport Loss for Learning Deep Neural Networks under Label Noise in Remote Sensing Images
• Iterative Deep Learning for Network Topology Extraction
• End to End Trainable Active Contours via Differentiable Rendering
• DeLS-3D: Deep Localization and Segmentation with a 3D Semantic Map
• SurfConv: Bridging 3D and 2D Convolution for RGBD Images
• Learning Local Features with Context Aggregation for Visual Localization
• Fine-Grained Vehicle Perception via 3D Part-Guided Visual Data Augmentation