SceneNet RGB-D: 5M Photorealistic Images of Synthetic Indoor Trajectories with Ground Truth

Abstract

We introduce SceneNet RGB-D, expanding the previous work of SceneNet to enable large scale photorealistic rendering of indoor scene trajectories. It provides pixel-perfect ground truth for scene understanding problems such as semantic segmentation, instance segmentation, and object detection, and also for geometric computer vision problems such as optical flow, depth estimation, camera pose estimation, and 3D reconstruction. Random sampling permits virtually unlimited scene configurations, and here we provide a set of 5M rendered RGB-D images from over 15K trajectories in synthetic layouts with random but physically simulated object poses. Each layout also has random lighting, camera trajectories, and textures. The scale of this dataset is well suited for pre-training data-driven computer vision techniques from scratch with RGB-D inputs, which previously has been limited by relatively small labelled datasets in NYUv2 and SUN RGB-D. It also provides a basis for investigating 3D scene labelling tasks by providing perfect camera poses and depth data as proxy for a SLAM system. We host the dataset at http://robotvault.bitbucket.io/scenenet-rgbd.html

References in corpus (5)

• FlowNet: Learning Optical Flow with Convolutional Networks
• Playing for Data: Ground Truth from Computer Games
• SceneNet: Understanding Real World Indoor Scenes With Synthetic Data
• Semantic Scene Completion from a Single Depth Image
• UnrealCV: Connecting Computer Vision to Unreal Engine

Cited by in corpus (22)

• Joint 2D-3D-Semantic Data for Indoor Scene Understanding
• AI2-THOR: An Interactive 3D Environment for Visual AI
• Indoor Scene Understanding in 2.5/3D for Autonomous Agents: A Survey
• Single Image Depth Estimation: An Overview
• InteriorNet: Mega-scale Multi-sensor Photo-realistic Indoor Scenes Dataset
• Peeking Behind Objects: Layered Depth Prediction from a Single Image
• UnrealText: Synthesizing Realistic Scene Text Images from the Unreal World
• Towards Semantic Segmentation of Urban-Scale 3D Point Clouds: A Dataset, Benchmarks and Challenges
• The AdobeIndoorNav Dataset: Towards Deep Reinforcement Learning based Real-world Indoor Robot Visual Navigation
• Are We Ready for Service Robots? The OpenLORIS-Scene Datasets for Lifelong SLAM
• Unbiasing Semantic Segmentation For Robot Perception using Synthetic Data Feature Transfer
• Adversarial Attacks Beyond the Image Space
• On The State of Data In Computer Vision: Human Annotations Remain Indispensable for Developing Deep Learning Models
• SynthText3D: Synthesizing Scene Text Images from 3D Virtual Worlds
• Transferable Active Grasping and Real Embodied Dataset
• Training Object Detectors on Synthetic Images Containing Reflecting Materials
• Learning Where to Look: Data-Driven Viewpoint Set Selection for 3D Scenes
• Learning Single-Image Depth from Videos using Quality Assessment Networks
• OASIS: A Large-Scale Dataset for Single Image 3D in the Wild
• What Synthesis is Missing: Depth Adaptation Integrated with Weak Supervision for Indoor Scene Parsing
• Learning Topology from Synthetic Data for Unsupervised Depth Completion
• Learning to Reconstruct and Segment 3D Objects