Sim4CV: A Photo-Realistic Simulator for Computer Vision Applications

Abstract

We present a photo-realistic training and evaluation simulator (Sim4CV) with extensive applications across various fields of computer vision. Built on top of the Unreal Engine, the simulator integrates full featured physics based cars, unmanned aerial vehicles (UAVs), and animated human actors in diverse urban and suburban 3D environments. We demonstrate the versatility of the simulator with two case studies: autonomous UAV-based tracking of moving objects and autonomous driving using supervised learning. The simulator fully integrates both several state-of-the-art tracking algorithms with a benchmark evaluation tool and a deep neural network (DNN) architecture for training vehicles to drive autonomously. It generates synthetic photo-realistic datasets with automatic ground truth annotations to easily extend existing real-world datasets and provides extensive synthetic data variety through its ability to reconfigure synthetic worlds on the fly using an automatic world generation tool. The supplementary video can be viewed a https://youtu.be/SqAxzsQ7qUU

Published at the International Journal of Computer Vision (IJCV), 2018

References in corpus (9)

• End to End Learning for Self-Driving Cars
• Beyond Correlation Filters: Learning Continuous Convolution Operators for Visual Tracking
• Learning Spatially Regularized Correlation Filters for Visual Tracking
• Asynchronous Methods for Deep Reinforcement Learning
• Virtual Worlds as Proxy for Multi-Object Tracking Analysis
• CARLA: An Open Urban Driving Simulator
• Playing for Data: Ground Truth from Computer Games
• Deep Neural Network for Real-Time Autonomous Indoor Navigation
• Learning Physical Intuition of Block Towers by Example

Cited by in corpus (20)

• AADS: Augmented Autonomous Driving Simulation using Data-driven Algorithms
• Auxiliary Tasks in Multi-task Learning
• Augmented LiDAR Simulator for Autonomous Driving
• MINOS: Multimodal Indoor Simulator for Navigation in Complex Environments
• Autonomous Drone Racing: A Survey
• Choose Your Simulator Wisely: A Review on Open-source Simulators for Autonomous Driving
• FusionPlanner: A Multi-task Motion Planner for Mining Trucks via Multi-sensor Fusion
• Flightmare: A Flexible Quadrotor Simulator
• BioDrone: A Bionic Drone-based Single Object Tracking Benchmark for Robust Vision
• SADA: Semantic Adversarial Diagnostic Attacks for Autonomous Applications
• Learning To Simulate
• Can Synthetic Faces Undo the Damage of Dataset Bias to Face Recognition and Facial Landmark Detection?
• Improving the Generalization of End-to-End Driving through Procedural Generation
• Synthetic Data-based Detection of Zebras in Drone Imagery
• SUMMIT: A Simulator for Urban Driving in Massive Mixed Traffic
• Simulating Autonomous Driving in Massive Mixed Urban Traffic
• Teaching UAVs to Race: End-to-End Regression of Agile Controls in Simulation
• Pose Estimation for Texture-less Shiny Objects in a Single RGB Image Using Synthetic Training Data
• Fine-Grained Vehicle Perception via 3D Part-Guided Visual Data Augmentation
• A Neural-Symbolic Framework for Mental Simulation