Hard Pixel Mining for Depth Privileged Semantic Segmentation

Abstract

Semantic segmentation has achieved remarkable progress but remains challenging due to the complex scene, object occlusion, and so on. Some research works have attempted to use extra information such as a depth map to help RGB based semantic segmentation because the depth map could provide complementary geometric cues. However, due to the inaccessibility of depth sensors, depth information is usually unavailable for the test images. In this paper, we leverage only the depth of training images as the privileged information to mine the hard pixels in semantic segmentation, in which depth information is only available for training images but not available for test images. Specifically, we propose a novel Loss Weight Module, which outputs a loss weight map by employing two depth-related measurements of hard pixels: Depth Prediction Error and Depthaware Segmentation Error. The loss weight map is then applied to segmentation loss, with the goal of learning a more robust model by paying more attention to the hard pixels. Besides, we also explore a curriculum learning strategy based on the loss weight map. Meanwhile, to fully mine the hard pixels on different scales, we apply our loss weight module to multi-scale side outputs. Our hard pixels mining method achieves the state-of-the-art results on two benchmark datasets, and even outperforms the methods which need depth input during testing.

arXiv comment13 pages

References in corpus (16)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• Rethinking Atrous Convolution for Semantic Image Segmentation
• Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials
• Conditional Random Fields as Recurrent Neural Networks
• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• Curriculum Domain Adaptation for Semantic Segmentation of Urban Scenes
• Convolutional Feature Masking for Joint Object and Stuff Segmentation
• Semantic Image Segmentation via Deep Parsing Network
• Unifying distillation and privileged information
• BoxSup: Exploiting Bounding Boxes to Supervise Convolutional Networks for Semantic Segmentation
• Light-Weight RefineNet for Real-Time Semantic Segmentation
• Depth Adaptive Deep Neural Network for Semantic Segmentation
• ACNet: Attention Based Network to Exploit Complementary Features for RGBD Semantic Segmentation
• Multiple Instance Curriculum Learning for Weakly Supervised Object Detection
• Transfer Hashing with Privileged Information