SemanticFusion: Dense 3D Semantic Mapping with Convolutional Neural Networks

Abstract

Ever more robust, accurate and detailed mapping using visual sensing has proven to be an enabling factor for mobile robots across a wide variety of applications. For the next level of robot intelligence and intuitive user interaction, maps need extend beyond geometry and appearence - they need to contain semantics. We address this challenge by combining Convolutional Neural Networks (CNNs) and a state of the art dense Simultaneous Localisation and Mapping (SLAM) system, ElasticFusion, which provides long-term dense correspondence between frames of indoor RGB-D video even during loopy scanning trajectories. These correspondences allow the CNN's semantic predictions from multiple view points to be probabilistically fused into a map. This not only produces a useful semantic 3D map, but we also show on the NYUv2 dataset that fusing multiple predictions leads to an improvement even in the 2D semantic labelling over baseline single frame predictions. We also show that for a smaller reconstruction dataset with larger variation in prediction viewpoint, the improvement over single frame segmentation increases. Our system is efficient enough to allow real-time interactive use at frame-rates of approximately 25Hz.

References in corpus (5)

• Caffe: Convolutional Architecture for Fast Feature Embedding
• Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials
• Learning Deconvolution Network for Semantic Segmentation
• Indoor Semantic Segmentation using depth information
• SceneNet: Understanding Real World Indoor Scenes With Synthetic Data

Cited by in corpus (14)

• ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes
• Panoptic Multi-TSDFs: a Flexible Representation for Online Multi-resolution Volumetric Mapping and Long-term Dynamic Scene Consistency
• FutureMapping: The Computational Structure of Spatial AI Systems
• Multi-View Deep Learning for Consistent Semantic Mapping with RGB-D Cameras
• ILabel: Interactive Neural Scene Labelling
• Dual Quadrics from Object Detection BoundingBoxes as Landmark Representations in SLAM
• Fusion++: Volumetric Object-Level SLAM
• Unbiasing Semantic Segmentation For Robot Perception using Synthetic Data Feature Transfer
• Pointwise Convolutional Neural Networks
• DA-RNN: Semantic Mapping with Data Associated Recurrent Neural Networks
• Visual-Inertial-Semantic Scene Representation for 3-D Object Detection
• STD2P: RGBD Semantic Segmentation Using Spatio-Temporal Data-Driven Pooling
• Semantic Dense Reconstruction with Consistent Scene Segments
• A Data-driven Prior on Facet Orientation for Semantic Mesh Labeling