Depth-aware CNN for RGB-D Segmentation

Abstract

Convolutional neural networks (CNN) are limited by the lack of capability to handle geometric information due to the fixed grid kernel structure. The availability of depth data enables progress in RGB-D semantic segmentation with CNNs. State-of-the-art methods either use depth as additional images or process spatial information in 3D volumes or point clouds. These methods suffer from high computation and memory cost. To address these issues, we present Depth-aware CNN by introducing two intuitive, flexible and effective operations: depth-aware convolution and depth-aware average pooling. By leveraging depth similarity between pixels in the process of information propagation, geometry is seamlessly incorporated into CNN. Without introducing any additional parameters, both operators can be easily integrated into existing CNNs. Extensive experiments and ablation studies on challenging RGB-D semantic segmentation benchmarks validate the effectiveness and flexibility of our approach.

References in corpus (3)

• Joint 2D-3D-Semantic Data for Indoor Scene Understanding
• Indoor Semantic Segmentation using depth information
• Scene Labeling using Gated Recurrent Units with Explicit Long Range Conditioning

Cited by in corpus (6)

• SNE-RoadSeg: Incorporating Surface Normal Information into Semantic Segmentation for Accurate Freespace Detection
• GeoConv: Geodesic Guided Convolution for Facial Action Unit Recognition
• ACNet: Attention Based Network to Exploit Complementary Features for RGBD Semantic Segmentation
• Real-time Fusion Network for RGB-D Semantic Segmentation Incorporating Unexpected Obstacle Detection for Road-driving Images
• Semantic Road Layout Understanding by Generative Adversarial Inpainting
• Fast Generation of High Fidelity RGB-D Images by Deep-Learning with Adaptive Convolution