Matterport3D: Learning from RGB-D Data in Indoor Environments

Abstract

Access to large, diverse RGB-D datasets is critical for training RGB-D scene understanding algorithms. However, existing datasets still cover only a limited number of views or a restricted scale of spaces. In this paper, we introduce Matterport3D, a large-scale RGB-D dataset containing 10,800 panoramic views from 194,400 RGB-D images of 90 building-scale scenes. Annotations are provided with surface reconstructions, camera poses, and 2D and 3D semantic segmentations. The precise global alignment and comprehensive, diverse panoramic set of views over entire buildings enable a variety of supervised and self-supervised computer vision tasks, including keypoint matching, view overlap prediction, normal prediction from color, semantic segmentation, and region classification.

References in corpus (9)

• Joint 2D-3D-Semantic Data for Indoor Scene Understanding
• ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes
• A Large Dataset of Object Scans
• LIFT: Learned Invariant Feature Transform
• SceneNet: Understanding Real World Indoor Scenes With Synthetic Data
• 3DMatch: Learning Local Geometric Descriptors from RGB-D Reconstructions
• Semantic Scene Completion from a Single Depth Image
• Deep unsupervised learning through spatial contrasting
• Physically-Based Rendering for Indoor Scene Understanding Using Convolutional Neural Networks

Cited by in corpus (151)

• On Evaluation of Embodied Navigation Agents
• AI2-THOR: An Interactive 3D Environment for Visual AI
• Cross-view Semantic Segmentation for Sensing Surroundings
• Object Goal Navigation using Goal-Oriented Semantic Exploration
• How Much Can CLIP Benefit Vision-and-Language Tasks?
• Indoor Scene Understanding in 2.5/3D for Autonomous Agents: A Survey
• Vision-and-Dialog Navigation
• HoME: a Household Multimodal Environment
• Language-guided Navigation via Cross-Modal Grounding and Alternate Adversarial Learning
• DIODE: A Dense Indoor and Outdoor DEpth Dataset
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• The StreetLearn Environment and Dataset
• Deep Learning Based 3D Segmentation: A Survey
• ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scans
• iGibson 1.0: a Simulation Environment for Interactive Tasks in Large Realistic Scenes
• Benchmarking Classic and Learned Navigation in Complex 3D Environments
• Embodied Visual Navigation with Automatic Curriculum Learning in Real Environments
• iPlanner: Imperative Path Planning
• Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation
• VRKitchen: an Interactive 3D Virtual Environment for Task-oriented Learning
• 3D Scene Geometry Estimation from 360${}^{\circ }$ Imagery: A Survey
• General Evaluation for Instruction Conditioned Navigation using Dynamic Time Warping
• Optimization and Manipulation of Contextual Mutual Spaces for Multi-User Virtual and Augmented Reality Interaction
• Structured3D: A Large Photo-realistic Dataset for Structured 3D Modeling
• HoliCity: A City-Scale Data Platform for Learning Holistic 3D Structures
• A Survey of Embodied AI: From Simulators to Research Tasks
• Semantic Visual Navigation by Watching YouTube Videos
• The AdobeIndoorNav Dataset: Towards Deep Reinforcement Learning based Real-world Indoor Robot Visual Navigation
• Hierarchical Topometric Representation of 3D Robotic Maps
• Scan2CAD: Learning CAD Model Alignment in RGB-D Scans
• Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigation
• Deep Depth Completion of a Single RGB-D Image
• Improving Target-driven Visual Navigation with Attention on 3D Spatial Relationships
• Language Tasks and Language Games: On Methodology in Current Natural Language Processing Research
• Retouchdown: Adding Touchdown to StreetLearn as a Shareable Resource for Language Grounding Tasks in Street View
• Language and Visual Entity Relationship Graph for Agent Navigation
• 3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans
• Embodied Vision-and-Language Navigation with Dynamic Convolutional Filters
• Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Tasks
• Learning models for visual 3D localization with implicit mapping
• 3D-FUTURE: 3D Furniture shape with TextURE
• To Learn or Not to Learn: Analyzing the Role of Learning for Navigation in Virtual Environments
• Auxiliary Tasks and Exploration Enable ObjectNav
• Evolving Graphical Planner: Contextual Global Planning for Vision-and-Language Navigation
• RMM: A Recursive Mental Model for Dialog Navigation
• The Regretful Agent: Heuristic-Aided Navigation through Progress Estimation
• Move to See Better: Self-Improving Embodied Object Detection
• REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments
• Robust Navigation with Language Pretraining and Stochastic Sampling
• Panoramic Depth Estimation via Supervised and Unsupervised Learning in Indoor Scenes
• Kaleido-BERT: Vision-Language Pre-training on Fashion Domain
• Learning Navigation Subroutines from Egocentric Videos
• SceneGen: Generative Contextual Scene Augmentation using Scene Graph Priors
• Convolutional Occupancy Networks
• Are You Looking? Grounding to Multiple Modalities in Vision-and-Language Navigation
• Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation
• 3D Shape Synthesis for Conceptual Design and Optimization Using Variational Autoencoders
• Learning Affordance Landscapes for Interaction Exploration in 3D Environments
• Im2Pano3D: Extrapolating 360 Structure and Semantics Beyond the Field of View
• RGBD Based Dimensional Decomposition Residual Network for 3D Semantic Scene Completion
• RGB-D Odometry and SLAM
• Tactical Rewind: Self-Correction via Backtracking in Vision-and-Language Navigation
• Synthetic 3D Data Generation Pipeline for Geometric Deep Learning in Architecture
• Simultaneous Mapping and Target Driven Navigation
• Multi-View Learning for Vision-and-Language Navigation
• SG-NN: Sparse Generative Neural Networks for Self-Supervised Scene Completion of RGB-D Scans
• COCO-GAN: Generation by Parts via Conditional Coordinating
• VisualEchoes: Spatial Image Representation Learning through Echolocation
• The ThreeDWorld Transport Challenge: A Visually Guided Task-and-Motion Planning Benchmark for Physically Realistic Embodied AI
• Why Build an Assistant in Minecraft?
• Grasping in the Wild:Learning 6DoF Closed-Loop Grasping from Low-Cost Demonstrations
• RarePlanes: Synthetic Data Takes Flight
• Learning to Visually Navigate in Photorealistic Environments Without any Supervision
• BabyWalk: Going Farther in Vision-and-Language Navigation by Taking Baby Steps
• LayoutMP3D: Layout Annotation of Matterport3D
• Neural Point Cloud Rendering via Multi-Plane Projection
• Spherical View Synthesis for Self-Supervised 360 Depth Estimation
• Manhattan Room Layout Reconstruction from a Single 360 image: A Comparative Study of State-of-the-art Methods
• Panoptic 3D Scene Reconstruction From a Single RGB Image
• Sub-Instruction Aware Vision-and-Language Navigation
• Arena: A General Evaluation Platform and Building Toolkit for Multi-Agent Intelligence
• Floor-SP: Inverse CAD for Floorplans by Sequential Room-wise Shortest Path
• Sim-Real Joint Reinforcement Transfer for 3D Indoor Navigation
• RevealNet: Seeing Behind Objects in RGB-D Scans
• An A* Curriculum Approach to Reinforcement Learning for RGBD Indoor Robot Navigation
• Efficient Semantic Scene Completion Network with Spatial Group Convolution
• Visual Question Answering on 360° Images
• 3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation
• Navigation Agents for the Visually Impaired: A Sidewalk Simulator and Experiments
• Learning Audio-Visual Dereverberation
• Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning
• SGoLAM: Simultaneous Goal Localization and Mapping for Multi-Object Goal Navigation
• SOON: Scenario Oriented Object Navigation with Graph-based Exploration
• Transferable Representation Learning in Vision-and-Language Navigation
• TextureNet: Consistent Local Parametrizations for Learning from High-Resolution Signals on Meshes
• Learning to Map for Active Semantic Goal Navigation
• SurfelGAN: Synthesizing Realistic Sensor Data for Autonomous Driving
• SSCNav: Confidence-Aware Semantic Scene Completion for Visual Semantic Navigation
• Predicting Performance of SLAM Algorithms
• Deep Surface Normal Estimation with Hierarchical RGB-D Fusion
• NeSF: Neural Semantic Fields for Generalizable Semantic Segmentation of 3D Scenes
• Learning Feature Descriptors using Camera Pose Supervision
• Vision-Dialog Navigation by Exploring Cross-modal Memory
• Dynamic Plane Convolutional Occupancy Networks
• SPSG: Self-Supervised Photometric Scene Generation from RGB-D Scans
• PointAR: Efficient Lighting Estimation for Mobile Augmented Reality
• GeoLayout: Geometry Driven Room Layout Estimation Based on Depth Maps of Planes
• Generative Language-Grounded Policy in Vision-and-Language Navigation with Bayes' Rule
• Restyling Data: Application to Unsupervised Domain Adaptation
• Take the Scenic Route: Improving Generalization in Vision-and-Language Navigation
• NavigationNet: A Large-scale Interactive Indoor Navigation Dataset
• MaAST: Map Attention with Semantic Transformersfor Efficient Visual Navigation
• ODE-CNN: Omnidirectional Depth Extension Networks
• Learning the Depths of Moving People by Watching Frozen People
• Assisted Perception: Optimizing Observations to Communicate State
• Spherical Transformer: Adapting Spherical Signal to CNNs
• Deep Visual MPC-Policy Learning for Navigation
• Depth Estimation via Affinity Learned with Convolutional Spatial Propagation Network
• BEyond observation: an approach for ObjectNav
• The Surprising Effectiveness of Visual Odometry Techniques for Embodied PointGoal Navigation
• Semantic Dense Reconstruction with Consistent Scene Segments
• Spherical Convolutional Neural Networks: Stability to Perturbations in SO(3)
• 3D Guided Weakly Supervised Semantic Segmentation
• Semantic Audio-Visual Navigation
• Visual Perception Generalization for Vision-and-Language Navigation via Meta-Learning
• PanGEA: The Panoramic Graph Environment Annotation Toolkit
• Hierarchical Cross-Modal Agent for Robotics Vision-and-Language Navigation
• Going Deeper with Lean Point Networks
• Active Imitation Learning from Multiple Non-Deterministic Teachers: Formulation, Challenges, and Algorithms
• Learning Single-Image Depth from Videos using Quality Assessment Networks
• Hierarchy Denoising Recursive Autoencoders for 3D Scene Layout Prediction
• $360^o$ Surface Regression with a Hyper-Sphere Loss
• Learning Equivariant Representations
• Towards Part-Based Understanding of RGB-D Scans
• AccSS3D: Accelerator for Spatially Sparse 3D DNNs
• Localising In Complex Scenes Using Balanced Adversarial Adaptation
• Learning to Explore by Reinforcement over High-Level Options
• Training Deep Neural Networks to Detect Repeatable 2D Features Using Large Amounts of 3D World Capture Data
• Planning with Submodular Objective Functions
• Natural Language for Human-Robot Collaboration: Problems Beyond Language Grounding
• ADeLA: Automatic Dense Labeling with Attention for Viewpoint Adaptation in Semantic Segmentation
• Online Unsupervised Learning of Visual Representations and Categories
• Distortion Reduction for Off-Center Perspective Projection of Panoramas
• Learning to Reconstruct and Understand Indoor Scenes from Sparse Views
• VSGM -- Enhance robot task understanding ability through visual semantic graph
• Indoor Panorama Planar 3D Reconstruction via Divide and Conquer
• MAOMaps: A Photo-Realistic Benchmark For vSLAM and Map Merging Quality Assessment
• Leveraging Semantics for Incremental Learning in Multi-Relational Embeddings
• Neural Illumination: Lighting Prediction for Indoor Environments
• Associative3D: Volumetric Reconstruction from Sparse Views
• Invisible Marker: Automatic Annotation of Segmentation Masks for Object Manipulation