Depth Adaptive Deep Neural Network for Semantic Segmentation

Abstract

In this work, we present the depth-adaptive deep neural network using a depth map for semantic segmentation. Typical deep neural networks receive inputs at the predetermined locations regardless of the distance from the camera. This fixed receptive field presents a challenge to generalize the features of objects at various distances in neural networks. Specifically, the predetermined receptive fields are too small at a short distance, and vice versa. To overcome this challenge, we develop a neural network which is able to adapt the receptive field not only for each layer but also for each neuron at the spatial location. To adjust the receptive field, we propose the depth-adaptive multiscale (DaM) convolution layer consisting of the adaptive perception neuron and the in-layer multiscale neuron. The adaptive perception neuron is to adjust the receptive field at each spatial location using the corresponding depth information. The in-layer multiscale neuron is to apply the different size of the receptive field at each feature space to learn features at multiple scales. The proposed DaM convolution is applied to two fully convolutional neural networks. We demonstrate the effectiveness of the proposed neural networks on the publicly available RGB-D dataset for semantic segmentation and the novel hand segmentation dataset for hand-object interaction. The experimental results show that the proposed method outperforms the state-of-the-art methods without any additional layers or pre/post-processing.

IEEE Transactions on Multimedia, 2018

References in corpus (7)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs
• Fully Convolutional Networks for Semantic Segmentation
• Conditional Random Fields as Recurrent Neural Networks
• The Cityscapes Dataset for Semantic Urban Scene Understanding
• 3D Object Reconstruction from Hand-Object Interactions

Cited by in corpus (13)

• FECANet: Boosting Few-Shot Semantic Segmentation with Feature-Enhanced Context-Aware Network
• Saliency Guided Inter- and Intra-Class Relation Constraints for Weakly Supervised Semantic Segmentation
• Learning to Measure Change: Fully Convolutional Siamese Metric Networks for Scene Change Detection
• Random Forest with Learned Representations for Semantic Segmentation
• Pixel-Level Clustering Network for Unsupervised Image Segmentation
• Enhancing Underexposed Photos using Perceptually Bidirectional Similarity
• Self-Ensembling GAN for Cross-Domain Semantic Segmentation
• Hand Segmentation for Hand-Object Interaction from Depth map
• Deep Semantic Parsing of Freehand Sketches with Homogeneous Transformation, Soft-Weighted Loss, and Staged Learning
• Multi-Modal Attention-based Fusion Model for Semantic Segmentation of RGB-Depth Images
• Hard Pixel Mining for Depth Privileged Semantic Segmentation
• Cross-Image Region Mining with Region Prototypical Network for Weakly Supervised Segmentation
• Malleable 2.5D Convolution: Learning Receptive Fields along the Depth-axis for RGB-D Scene Parsing