SurfelGAN: Synthesizing Realistic Sensor Data for Autonomous Driving
arXiv:2005.03844
Abstract
Autonomous driving system development is critically dependent on the ability to replay complex and diverse traffic scenarios in simulation. In such scenarios, the ability to accurately simulate the vehicle sensors such as cameras, lidar or radar is essential. However, current sensor simulators leverage gaming engines such as Unreal or Unity, requiring manual creation of environments, objects and material properties. Such approaches have limited scalability and fail to produce realistic approximations of camera, lidar, and radar data without significant additional work. In this paper, we present a simple yet effective approach to generate realistic scenario sensor data, based only on a limited amount of lidar and camera data collected by an autonomous vehicle. Our approach uses texture-mapped surfels to efficiently reconstruct the scene from an initial vehicle pass or set of passes, preserving rich information about object 3D geometry and appearance, as well as the scene conditions. We then leverage a SurfelGAN network to reconstruct realistic camera images for novel positions and orientations of the self-driving vehicle and moving objects in the scene. We demonstrate our approach on the Waymo Open Dataset and show that it can synthesize realistic camera data for simulated scenarios. We also create a novel dataset that contains cases in which two self-driving vehicles observe the same scene at the same time. We use this dataset to provide additional evaluation and demonstrate the usefulness of our SurfelGAN model.
References in corpus (15)
- Adam: A Method for Stochastic Optimization
- Generative Adversarial Networks: An Overview
- Spectral Normalization for Generative Adversarial Networks
- Progressive Growing of GANs for Improved Quality, Stability, and Variation
- Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net
- Matterport3D: Learning from RGB-D Data in Indoor Environments
- CARLA: An Open Urban Driving Simulator
- Building Generalizable Agents with a Realistic and Rich 3D Environment
- Scalability in Perception for Autonomous Driving: Waymo Open Dataset
- AADS: Augmented Autonomous Driving Simulation using Data-driven Algorithms
- MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction
- Video-to-Video Synthesis
- ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst
- From Points to Parts: 3D Object Detection from Point Cloud with Part-aware and Part-aggregation Network
- HoME: a Household Multimodal Environment