GeoSim: Realistic Video Simulation via Geometry-Aware Composition for Self-Driving

Abstract

Scalable sensor simulation is an important yet challenging open problem for safety-critical domains such as self-driving. Current works in image simulation either fail to be photorealistic or do not model the 3D environment and the dynamic objects within, losing high-level control and physical realism. In this paper, we present GeoSim, a geometry-aware image composition process which synthesizes novel urban driving scenarios by augmenting existing images with dynamic objects extracted from other scenes and rendered at novel poses. Towards this goal, we first build a diverse bank of 3D objects with both realistic geometry and appearance from sensor data. During simulation, we perform a novel geometry-aware simulation-by-composition procedure which 1) proposes plausible and realistic object placements into a given scene, 2) render novel views of dynamic objects from the asset bank, and 3) composes and blends the rendered image segments. The resulting synthetic images are realistic, traffic-aware, and geometrically consistent, allowing our approach to scale to complex use cases. We demonstrate two such important applications: long-range realistic video simulation across multiple camera sensors, and synthetic data generation for data augmentation on downstream segmentation tasks. Please check https://tmux.top/publication/geosim/ for high-resolution video results.

arXiv commentAccepted by CVPR 2021 as Oral

References in corpus (14)

• Rethinking Atrous Convolution for Semantic Image Segmentation
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• AI2-THOR: An Interactive 3D Environment for Visual AI
• Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer
• AADS: Augmented Autonomous Driving Simulation using Data-driven Algorithms
• Augmented LiDAR Simulator for Autonomous Driving
• MINOS: Multimodal Indoor Simulator for Navigation in Complex Environments
• Learning Hierarchical Semantic Image Manipulation through Structured Representations
• Context-Aware Synthesis and Placement of Object Instances
• Beyond Grand Theft Auto V for Training, Testing and Enhancing Deep Learning in Self Driving Cars
• ST-GAN: Spatial Transformer Generative Adversarial Networks for Image Compositing
• RELATE: Physically Plausible Multi-Object Scene Synthesis Using Structured Latent Spaces
• Image GANs meet Differentiable Rendering for Inverse Graphics and Interpretable 3D Neural Rendering
• SurfelGAN: Synthesizing Realistic Sensor Data for Autonomous Driving