3D Gated Recurrent Fusion for Semantic Scene Completion
arXiv:2002.07269
Abstract
This paper tackles the problem of data fusion in the semantic scene completion (SSC) task, which can simultaneously deal with semantic labeling and scene completion. RGB images contain texture details of the object(s) which are vital for semantic scene understanding. Meanwhile, depth images capture geometric clues of high relevance for shape completion. Using both RGB and depth images can further boost the accuracy of SSC over employing one modality in isolation. We propose a 3D gated recurrent fusion network (GRFNet), which learns to adaptively select and fuse the relevant information from depth and RGB by making use of the gate and memory modules. Based on the single-stage fusion, we further propose a multi-stage fusion strategy, which could model the correlations among different stages within the network. Extensive experiments on two benchmark datasets demonstrate the superior performance and the effectiveness of the proposed GRFNet for data fusion in SSC. Code will be made available.
13 pages
References in corpus (11)
- Two-Stream Convolutional Networks for Action Recognition in Videos
- Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
- Attention-Based Models for Speech Recognition
- Character-Aware Neural Language Models
- Indoor Semantic Segmentation using depth information
- Delving Deeper into Convolutional Networks for Learning Video Representations
- Scalable Place Recognition Under Appearance Change for Autonomous Driving
- Unsupervised RGBD Video Object Segmentation Using GANs
- RGB-D Based Action Recognition with Light-weight 3D Convolutional Networks
- Adaptive Fusion for RGB-D Salient Object Detection
- Semantic Augmented Reality Environment with Material-Aware Physical Interactions
Cited by in corpus (3)
- Sparse Single Sweep LiDAR Point Cloud Segmentation via Learning Contextual Shape Priors from Scene Completion
- Gated Recurrent Fusion with Joint Training Framework for Robust End-to-End Speech Recognition
- IMENet: Joint 3D Semantic Scene Completion and 2D Semantic Segmentation through Iterative Mutual Enhancement