Configurable 3D Scene Synthesis and 2D Image Rendering with Per-Pixel Ground Truth using Stochastic Grammars

Abstract

We propose a systematic learning-based approach to the generation of massive quantities of synthetic 3D scenes and arbitrary numbers of photorealistic 2D images thereof, with associated ground truth information, for the purposes of training, benchmarking, and diagnosing learning-based computer vision and robotics algorithms. In particular, we devise a learning-based pipeline of algorithms capable of automatically generating and rendering a potentially infinite variety of indoor scenes by using a stochastic grammar, represented as an attributed Spatial And-Or Graph, in conjunction with state-of-the-art physics-based rendering. Our pipeline is capable of synthesizing scene layouts with high diversity, and it is configurable inasmuch as it enables the precise customization and control of important attributes of the generated scenes. It renders photorealistic RGB images of the generated scenes while automatically synthesizing detailed, per-pixel ground truth data, including visible surface depth and normal, object identity, and material information (detailed to object parts), as well as environments (e.g., illuminations and camera viewpoints). We demonstrate the value of our synthesized dataset, by improving performance in certain machine-learning-based scene understanding tasks--depth and surface normal prediction, semantic segmentation, reconstruction, etc.--and by providing benchmarks for and diagnostics of trained models by modifying object attributes and scene properties in a controllable manner.

Accepted in IJCV 2018

References in corpus (6)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Unsupervised Domain Adaptation by Backpropagation
• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• Learning from Synthetic Humans
• MoCap-guided Data Augmentation for 3D Pose Estimation in the Wild
• Bayesian Multitask Learning with Latent Hierarchies

Cited by in corpus (8)

• Scenic: A Language for Scenario Specification and Scene Generation
• Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense
• Holistic++ Scene Understanding: Single-view 3D Holistic Scene Parsing and Human Pose Estimation with Human-Object Interaction and Physical Commonsense
• Can Synthetic Faces Undo the Damage of Dataset Bias to Face Recognition and Facial Landmark Detection?
• Intelligent Generation of Graphical Game Assets: A Conceptual Framework and Systematic Review of the State of the Art
• Learning to Generate Synthetic 3D Training Data through Hybrid Gradient
• AGENT: A Benchmark for Core Psychological Reasoning
• MINERVAS: Massive INterior EnviRonments VirtuAl Synthesis