Virtual Worlds as Proxy for Multi-Object Tracking Analysis
arXiv:1605.06457
Abstract
Modern computer vision algorithms typically require expensive data acquisition and accurate manual labeling. In this work, we instead leverage the recent progress in computer graphics to generate fully labeled, dynamic, and photo-realistic proxy virtual worlds. We propose an efficient real-to-virtual world cloning method, and validate our approach by building and publicly releasing a new video dataset, called Virtual KITTI (see http://www.xrce.xerox.com/Research-Development/Computer-Vision/Proxy-Virtual-Worlds), automatically labeled with accurate ground truth for object detection, tracking, scene and instance segmentation, depth, and optical flow. We provide quantitative experimental evidence suggesting that (i) modern deep learning algorithms pre-trained on real data behave similarly in real and virtual worlds, and (ii) pre-training on virtual data improves performance. As the gap between real and virtual worlds is small, virtual worlds enable measuring the impact of various weather and imaging conditions on recognition performance, all other things being equal. We show these factors may affect drastically otherwise high-performing deep models for tracking.
CVPR 2016, Virtual KITTI dataset download at http://www.xrce.xerox.com/Research-Development/Computer-Vision/Proxy-Virtual-Worlds
References in corpus (4)
Cited by in corpus (123)
- Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges
- Learning from Synthetic Humans
- AI2-THOR: An Interactive 3D Environment for Visual AI
- Domain Adaptation for Visual Applications: A Comprehensive Survey
- CARLA: An Open Urban Driving Simulator
- AADS: Augmented Autonomous Driving Simulation using Data-driven Algorithms
- Sim4CV: A Photo-Realistic Simulator for Computer Vision Applications
- Modeling Surface Appearance from a Single Photograph using Self-augmented Convolutional Neural Networks
- Augmented LiDAR Simulator for Autonomous Driving
- Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning
- Uncertainty-Aware Consistency Regularization for Cross-Domain Semantic Segmentation
- FieldSAFE: Dataset for Obstacle Detection in Agriculture
- How much real data do we actually need: Analyzing object detection performance using synthetic and real data
- Training Deep Face Recognition Systems with Synthetic Data
- Joint Monocular 3D Vehicle Detection and Tracking
- Procedural Modeling and Physically Based Rendering for Synthetic Data Generation in Automotive Applications
- Extending Maps with Semantic and Contextual Object Information for Robot Navigation: a Learning-Based Framework using Visual and Depth Cues
- Real-time Joint Object Detection and Semantic Segmentation Network for Automated Driving
- SAFENet: Self-Supervised Monocular Depth Estimation with Semantic-Aware Feature Extraction
- Unsupervised Monocular Depth Estimation with Left-Right Consistency
- Drop to Adapt: Learning Discriminative Features for Unsupervised Domain Adaptation
- Back to Basics: Unsupervised Learning of Optical Flow via Brightness Constancy and Motion Smoothness
- AuxNet: Auxiliary tasks enhanced Semantic Segmentation for Automated Driving
- Learning Guided Convolutional Network for Depth Completion
- How to Train a CAT: Learning Canonical Appearance Transformations for Direct Visual Localization Under Illumination Change
- Image to Image Translation for Domain Adaptation
- Mixing Real and Synthetic Data to Enhance Neural Network Training -- A Review of Current Approaches
- BlendedMVS: A Large-scale Dataset for Generalized Multi-view Stereo Networks
- SceneFlowFields++: Multi-frame Matching, Visibility Prediction, and Robust Interpolation for Scene Flow Estimation
- Survey on Reliable Deep Learning-Based Person Re-Identification Models: Are We There Yet?
- Learning Human Optical Flow
- Metamorphic Testing for Object Detection Systems
- Full-Resolution Residual Networks for Semantic Segmentation in Street Scenes
- Keep it Simple: Image Statistics Matching for Domain Adaptation
- VirtualHome: Simulating Household Activities via Programs
- SimAug: Learning Robust Representations from Simulation for Trajectory Prediction
- UnrealCV: Connecting Computer Vision to Unreal Engine
- Warping of Radar Data into Camera Image for Cross-Modal Supervision in Automotive Applications
- Dissecting Person Re-identification from the Viewpoint of Viewpoint
- Unbiasing Semantic Segmentation For Robot Perception using Synthetic Data Feature Transfer
- IterGANs: Iterative GANs to Learn and Control 3D Object Transformation
- Active Perception using Light Curtains for Autonomous Driving
- Monocular Dense 3D Reconstruction of a Complex Dynamic Scene from Two Perspective Frames
- SharinGAN: Combining Synthetic and Real Data for Unsupervised Geometry Estimation
- Geometry-Aware Symmetric Domain Adaptation for Monocular Depth Estimation
- Dense Depth Posterior (DDP) from Single Image and Sparse Range
- T2Net: Synthetic-to-Realistic Translation for Solving Single-Image Depth Estimation Tasks
- UnrealStereo: Controlling Hazardous Factors to Analyze Stereo Vision
- RarePlanes: Synthetic Data Takes Flight
- Deep Supervision with Shape Concepts for Occlusion-Aware 3D Object Parsing
- Learning Depth via Leveraging Semantics: Self-supervised Monocular Depth Estimation with Both Implicit and Explicit Semantic Guidance
- End-to-end depth from motion with stabilized monocular videos
- The ParallelEye Dataset: Constructing Large-Scale Artificial Scenes for Traffic Vision Research
- Affordance Learning In Direct Perception for Autonomous Driving
- Spatial Fusion GAN for Image Synthesis
- A Curriculum Domain Adaptation Approach to the Semantic Segmentation of Urban Scenes
- Simultaneous Localization and Mapping with Dynamic Rigid Objects
- Weakly Supervised Silhouette-based Semantic Scene Change Detection
- MBAPose: Mask and Bounding-Box Aware Pose Estimation of Surgical Instruments with Photorealistic Domain Randomization
- Equivariant Maps for Hierarchical Structures
- TransFlow: Unsupervised Motion Flow by Joint Geometric and Pixel-level Estimation
- Image Classification for Arabic: Assessing the Accuracy of Direct English to Arabic Translations
- Semantic Nearest Neighbor Fields Monocular Edge Visual-Odometry
- A Fusion Approach for Multi-Frame Optical Flow Estimation
- Identifying Model Weakness with Adversarial Examiner
- Zoom and Learn: Generalizing Deep Stereo Matching to Novel Domains
- Multi-Resolution Graph Neural Network for Large-Scale Pointcloud Segmentation
- Cross-Domain Self-supervised Multi-task Feature Learning using Synthetic Imagery
- DF-Net: Unsupervised Joint Learning of Depth and Flow using Cross-Task Consistency
- Neural Network Generalization: The impact of camera parameters
- Leveraging synthetic imagery for collision-at-sea avoidance
- Automatic Generation of Dense Non-rigid Optical Flow
- PerMO: Perceiving More at Once from a Single Image for Autonomous Driving
- Synthetically Trained Icon Proposals for Parsing and Summarizing Infographics
- Learning Motion Patterns in Videos
- Efficient Realistic Data Generation Framework leveraging Deep Learning-based Human Digitization
- Training and Testing Object Detectors with Virtual Images
- Dynamic SLAM: The Need For Speed
- Soft Prototyping Camera Designs for Car Detection Based on a Convolutional Neural Network
- Learning Boost by Exploiting the Auxiliary Task in Multi-task Domain
- Creating a Large-scale Synthetic Dataset for Human Activity Recognition
- Learning to simulate complex scenes
- Optical Flow in Dense Foggy Scenes using Semi-Supervised Learning
- S-BEV: Semantic Birds-Eye View Representation for Weather and Lighting Invariant 3-DoF Localization
- S2R-DepthNet: Learning a Generalizable Depth-specific Structural Representation
- CRAVES: Controlling Robotic Arm with a Vision-based Economic System
- Continual Occlusions and Optical Flow Estimation
- Teaching UAVs to Race: End-to-End Regression of Agile Controls in Simulation
- SUB-Depth: Self-distillation and Uncertainty Boosting Self-supervised Monocular Depth Estimation
- Depth Completion via Deep Basis Fitting
- Virtual Testbed for Monocular Visual Navigation of Small Unmanned Aircraft Systems
- Restyling Data: Application to Unsupervised Domain Adaptation
- Volumetric Propagation Network: Stereo-LiDAR Fusion for Long-Range Depth Estimation
- Sim2Real for Self-Supervised Monocular Depth and Segmentation
- Supervized Segmentation with Graph-Structured Deep Metric Learning
- Novel Perception Algorithmic Framework For Object Identification and Tracking In Autonomous Navigation
- Cirrus: A Long-range Bi-pattern LiDAR Dataset
- EDEN: Multimodal Synthetic Dataset of Enclosed GarDEN Scenes
- Depth Map Estimation of Dynamic Scenes Using Prior Depth Information
- Unsupervised Visual Representation Learning by Tracking Patches in Video
- CGIntrinsics: Better Intrinsic Image Decomposition through Physically-Based Rendering
- Domain Adaptive Monocular Depth Estimation With Semantic Information
- Robust Ego and Object 6-DoF Motion Estimation and Tracking
- Unsupervised Domain Adaptation by Optical Flow Augmentation in Semantic Segmentation
- Car Pose in Context: Accurate Pose Estimation with Ground Plane Constraints
- Learning Rank Reduced Interpolation with Principal Component Analysis
- RGB-based Semantic Segmentation Using Self-Supervised Depth Pre-Training
- Object Detection on Single Monocular Images through Canonical Correlation Analysis
- Robust Monocular Edge Visual Odometry through Coarse-to-Fine Data Association
- Depth Edge Guided CNNs for Sparse Depth Upsampling
- Learning to Map Vehicles into Bird's Eye View
- Synthetic-to-Real Domain Adaptation for Lane Detection
- Do End-to-end Stereo Algorithms Under-utilize Information?
- Noisy-LSTM: Improving Temporal Awareness for Video Semantic Segmentation
- Place recognition survey: An update on deep learning approaches
- 3D Trajectory Reconstruction of Dynamic Objects Using Planarity Constraints
- CAR -- Cityscapes Attributes Recognition A Multi-category Attributes Dataset for Autonomous Vehicles
- Synthetic Data Are as Good as the Real for Association Knowledge Learning in Multi-object Tracking
- Learning color space adaptation from synthetic to real images of cirrus clouds
- On Offline Evaluation of Vision-based Driving Models
- Manual-Label Free 3D Detection via An Open-Source Simulator
- Automated Scene Flow Data Generation for Training and Verification
- Fine-Grained Vehicle Perception via 3D Part-Guided Visual Data Augmentation