Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentation

Abstract

Depth information has proven to be a useful cue in the semantic segmentation of RGB-D images for providing a geometric counterpart to the RGB representation. Most existing works simply assume that depth measurements are accurate and well-aligned with the RGB pixels and models the problem as a cross-modal feature fusion to obtain better feature representations to achieve more accurate segmentation. This, however, may not lead to satisfactory results as actual depth data are generally noisy, which might worsen the accuracy as the networks go deeper. In this paper, we propose a unified and efficient Cross-modality Guided Encoder to not only effectively recalibrate RGB feature responses, but also to distill accurate depth information via multiple stages and aggregate the two recalibrated representations alternatively. The key of the proposed architecture is a novel Separation-and-Aggregation Gating operation that jointly filters and recalibrates both representations before cross-modality aggregation. Meanwhile, a Bi-direction Multi-step Propagation strategy is introduced, on the one hand, to help to propagate and fuse information between the two modalities, and on the other hand, to preserve their specificity along the long-term propagation process. Besides, our proposed encoder can be easily injected into the previous encoder-decoder structures to boost their performance on RGB-D semantic segmentation. Our model outperforms state-of-the-arts consistently on both in-door and out-door challenging datasets. Code of this work is available at https://charlescxk.github.io/

Accepted by ECCV 2020

References in corpus (11)

• Rethinking Atrous Convolution for Semantic Image Segmentation
• Fully Convolutional Networks for Semantic Segmentation
• Squeeze-and-Excitation Networks
• OCNet: Object Context Network for Scene Parsing
• FastFCN: Rethinking Dilated Convolution in the Backbone for Semantic Segmentation
• Gated-SCNN: Gated Shape CNNs for Semantic Segmentation
• PAD-Net: Multi-Tasks Guided Prediction-and-Distillation Network for Simultaneous Depth Estimation and Scene Parsing
• RFBNet: Deep Multimodal Networks with Residual Fusion Blocks for RGB-D Semantic Segmentation
• Global Aggregation then Local Distribution in Fully Convolutional Networks
• SPGNet: Semantic Prediction Guidance for Scene Parsing
• ACNet: Attention Based Network to Exploit Complementary Features for RGBD Semantic Segmentation

Cited by in corpus (5)

• MobileSal: Extremely Efficient RGB-D Salient Object Detection
• Joint Implicit Image Function for Guided Depth Super-Resolution
• A Comprehensive Review of Modern Object Segmentation Approaches
• Global-Local Propagation Network for RGB-D Semantic Segmentation
• Encoder Fusion Network with Co-Attention Embedding for Referring Image Segmentation