RFBNet: Deep Multimodal Networks with Residual Fusion Blocks for RGB-D Semantic Segmentation

Abstract

RGB-D semantic segmentation methods conventionally use two independent encoders to extract features from the RGB and depth data. However, there lacks an effective fusion mechanism to bridge the encoders, for the purpose of fully exploiting the complementary information from multiple modalities. This paper proposes a novel bottom-up interactive fusion structure to model the interdependencies between the encoders. The structure introduces an interaction stream to interconnect the encoders. The interaction stream not only progressively aggregates modality-specific features from the encoders but also computes complementary features for them. To instantiate this structure, the paper proposes a residual fusion block (RFB) to formulate the interdependences of the encoders. The RFB consists of two residual units and one fusion unit with gate mechanism. It learns complementary features for the modality-specific encoders and extracts modality-specific features as well as cross-modal features. Based on the RFB, the paper presents the deep multimodal networks for RGB-D semantic segmentation called RFBNet. The experiments on two datasets demonstrate the effectiveness of modeling the interdependencies and that the RFBNet achieved state-of-the-art performance.

References in corpus (8)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Training Very Deep Networks
• Indoor Semantic Segmentation using depth information
• RedNet: Residual Encoder-Decoder Network for indoor RGB-D Semantic Segmentation
• Restricted Deformable Convolution based Road Scene Semantic Segmentation Using Surround View Cameras
• Fast Scene Understanding for Autonomous Driving
• Depth CNNs for RGB-D scene recognition: learning from scratch better than transferring from RGB-CNNs
• GFF: Gated Fully Fusion for Semantic Segmentation

Cited by in corpus (10)

• Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentation
• Siamese Network for RGB-D Salient Object Detection and Beyond
• Improving Multispectral Pedestrian Detection by Addressing Modality Imbalance Problems
• Real-time Fusion Network for RGB-D Semantic Segmentation Incorporating Unexpected Obstacle Detection for Road-driving Images
• 3D-to-2D Distillation for Indoor Scene Parsing
• Bidirectional Projection Network for Cross Dimension Scene Understanding
• Lightweight integration of 3D features to improve 2D image segmentation
• UNO: Uncertainty-aware Noisy-Or Multimodal Fusion for Unanticipated Input Degradation
• Deep feature selection-and-fusion for RGB-D semantic segmentation
• Evaluation of Multimodal Semantic Segmentation using RGB-D Data