Playing for Data: Ground Truth from Computer Games
arXiv:1608.02192
Abstract
Recent progress in computer vision has been driven by high-capacity models trained on large datasets. Unfortunately, creating large datasets with pixel-level labels has been extremely costly due to the amount of human effort required. In this paper, we present an approach to rapidly creating pixel-accurate semantic label maps for images extracted from modern computer games. Although the source code and the internal operation of commercial games are inaccessible, we show that associations between image patches can be reconstructed from the communication between the game and the graphics hardware. This enables rapid propagation of semantic labels within and across images synthesized by the game, with no access to the source code or the content. We validate the presented approach by producing dense pixel-level semantic annotations for 25 thousand images synthesized by a photorealistic open-world computer game. Experiments on semantic segmentation datasets show that using the acquired data to supplement real-world images significantly increases accuracy and that the acquired data enables reducing the amount of hand-labeled real-world data: models trained with game data and just 1/3 of the CamVid training set outperform models trained on the complete CamVid training set.
Accepted to the 14th European Conference on Computer Vision (ECCV 2016)
References in corpus (2)
Cited by in corpus (72)
- FCNs in the Wild: Pixel-level Adversarial and Constraint-based Adaptation
- CyCADA: Cycle-Consistent Adversarial Domain Adaptation
- GAN Augmentation: Augmenting Training Data using Generative Adversarial Networks
- Domain Adaptation for Visual Applications: A Comprehensive Survey
- CARLA: An Open Urban Driving Simulator
- Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World
- Sim4CV: A Photo-Realistic Simulator for Computer Vision Applications
- Automatic Model Based Dataset Generation for Fast and Accurate Crop and Weeds Detection
- Maximum Classifier Discrepancy for Unsupervised Domain Adaptation
- Learning Linear Transformations for Fast Arbitrary Style Transfer
- SceneNet RGB-D: 5M Photorealistic Images of Synthetic Indoor Trajectories with Ground Truth
- DSP: Dual Soft-Paste for Unsupervised Domain Adaptive Semantic Segmentation
- Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks
- PhotoShape: Photorealistic Materials for Large-Scale Shape Collections
- Domain Stylization: A Strong, Simple Baseline for Synthetic to Real Image Domain Adaptation
- Semantic-aware Grad-GAN for Virtual-to-Real Urban Scene Adaption
- Fully Convolutional Adaptation Networks for Semantic Segmentation
- SqueezeSeg: Convolutional Neural Nets with Recurrent CRF for Real-Time Road-Object Segmentation from 3D LiDAR Point Cloud
- Unsupervised Histopathology Image Synthesis
- Syn2Real: A New Benchmark forSynthetic-to-Real Visual Domain Adaptation
- A LiDAR Point Cloud Generator: from a Virtual World to Autonomous Driving
- DeepMVS: Learning Multi-view Stereopsis
- Rectifying Pseudo Label Learning via Uncertainty Estimation for Domain Adaptive Semantic Segmentation
- Video Propagation Networks
- Image to Image Translation for Domain Adaptation
- CEREALS - Cost-Effective REgion-based Active Learning for Semantic Segmentation
- Significance-aware Information Bottleneck for Domain Adaptive Semantic Segmentation
- No More Discrimination: Cross City Adaptation of Road Scene Segmenters
- All about Structure: Adapting Structural Information across Domains for Boosting Semantic Segmentation
- Effective Use of Synthetic Data for Urban Scene Semantic Segmentation
- TorontoCity: Seeing the World with a Million Eyes
- VirtualHome: Simulating Household Activities via Programs
- Adaptive Semantic Segmentation with a Strategic Curriculum of Proxy Labels
- Synthetic to Real Adaptation with Generative Correlation Alignment Networks
- Unsupervised Video-to-Video Translation
- ScaleNet: Guiding Object Proposal Generation in Supermarkets and Beyond
- ACE: Adapting to Changing Environments for Semantic Segmentation
- Adaptive Affinity Fields for Semantic Segmentation
- Unbiasing Semantic Segmentation For Robot Perception using Synthetic Data Feature Transfer
- Semi-parametric Image Synthesis
- LabelFusion: A Pipeline for Generating Ground Truth Labels for Real RGBD Data of Cluttered Scenes
- UnrealStereo: Controlling Hazardous Factors to Analyze Stereo Vision
- The ParallelEye Dataset: Constructing Large-Scale Artificial Scenes for Traffic Vision Research
- Semi-supervised Domain Adaptation based on Dual-level Domain Mixing for Semantic Segmentation
- Find Your Own Way: Weakly-Supervised Segmentation of Path Proposals for Urban Autonomy
- Deep Supervision with Shape Concepts for Occlusion-Aware 3D Object Parsing
- SAD-GAN: Synthetic Autonomous Driving using Generative Adversarial Networks
- On The State of Data In Computer Vision: Human Annotations Remain Indispensable for Developing Deep Learning Models
- BIM Hyperreality: Data Synthesis Using BIM and Hyperrealistic Rendering for Deep Learning
- Semantic Road Layout Understanding by Generative Adversarial Inpainting
- Cross-Domain Self-supervised Multi-task Feature Learning using Synthetic Imagery
- Physically-Based Rendering for Indoor Scene Understanding Using Convolutional Neural Networks
- Training and Testing Object Detectors with Virtual Images
- A Fully Convolutional Tri-branch Network (FCTN) for Domain Adaptation
- Learning Where to Look: Data-Driven Viewpoint Set Selection for 3D Scenes
- SUPER: A Novel Lane Detection System
- SyDog: A Synthetic Dog Dataset for Improved 2D Pose Estimation
- A Domain Agnostic Normalization Layer for Unsupervised Adversarial Domain Adaptation
- Enter the Matrix: Safely Interruptible Autonomous Systems via Virtualization
- Shape from Shading through Shape Evolution
- Canonical Correlation Analysis for Misaligned Satellite Image Change Detection
- A Brief Survey and an Application of Semantic Image Segmentation for Autonomous Driving
- A Dataset for Lane Instance Segmentation in Urban Environments
- Dynamic Adaptation on Non-Stationary Visual Domains
- CGIntrinsics: Better Intrinsic Image Decomposition through Physically-Based Rendering
- Leveraging Domain Adaptation for Low-Resource Geospatial Machine Learning
- Weakly-Supervised Cross-Domain Adaptation for Endoscopic Lesions Segmentation
- Soccer on Your Tabletop
- Multimodal Image Synthesis with Conditional Implicit Maximum Likelihood Estimation
- On Offline Evaluation of Vision-based Driving Models
- Learning to Map Vehicles into Bird's Eye View
- 3D Trajectory Reconstruction of Dynamic Objects Using Planarity Constraints