Deep Structured Scene Parsing by Learning with Image Descriptions

Abstract

This paper addresses a fundamental problem of scene understanding: How to parse the scene image into a structured configuration (i.e., a semantic object hierarchy with object interaction relations) that finely accords with human perception. We propose a deep architecture consisting of two networks: i) a convolutional neural network (CNN) extracting the image representation for pixelwise object labeling and ii) a recursive neural network (RNN) discovering the hierarchical object structure and the inter-object relations. Rather than relying on elaborative user annotations (e.g., manually labeling semantic maps and relations), we train our deep model in a weakly-supervised manner by leveraging the descriptive sentences of the training images. Specifically, we decompose each sentence into a semantic tree consisting of nouns and verb phrases, and facilitate these trees discovering the configurations of the training images. Once these scene configurations are determined, then the parameters of both the CNN and RNN are updated accordingly by back propagation. The entire model training is accomplished through an Expectation-Maximization method. Extensive experiments suggest that our model is capable of producing meaningful and structured scene configurations and achieving more favorable scene labeling performance on PASCAL VOC 2012 over other state-of-the-art weakly-supervised methods.

Discovering a semantic object hierarchy with object interaction relations (Publhised in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. (oral))

References in corpus (7)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Fully Convolutional Networks for Semantic Segmentation
• Fully Convolutional Multi-Class Multiple Instance Learning
• Semantic Image Segmentation via Deep Parsing Network
• Deep Visual-Semantic Alignments for Generating Image Descriptions
• Feedforward semantic segmentation with zoom-out features
• Deep Hierarchical Parsing for Semantic Segmentation

Cited by in corpus (5)

• Learning Deep Representations for Semantic Image Parsing: a Comprehensive Overview
• Deep Co-Space: Sample Mining Across Feature Transformation for Semi-Supervised Learning
• Human Pose Estimation from Depth Images via Inference Embedded Multi-task Learning
• Fully Convolutional Network for Melanoma Diagnostics
• Hierarchical Scene Parsing by Weakly Supervised Learning with Image Descriptions