Semantic Object Parsing with Graph LSTM

Abstract

By taking the semantic object parsing task as an exemplar application scenario, we propose the Graph Long Short-Term Memory (Graph LSTM) network, which is the generalization of LSTM from sequential data or multi-dimensional data to general graph-structured data. Particularly, instead of evenly and fixedly dividing an image to pixels or patches in existing multi-dimensional LSTM structures (e.g., Row, Grid and Diagonal LSTMs), we take each arbitrary-shaped superpixel as a semantically consistent node, and adaptively construct an undirected graph for each image, where the spatial relations of the superpixels are naturally used as edges. Constructed on such an adaptive graph topology, the Graph LSTM is more naturally aligned with the visual patterns in the image (e.g., object boundaries or appearance similarities) and provides a more economical information propagation route. Furthermore, for each optimization step over Graph LSTM, we propose to use a confidence-driven scheme to update the hidden and memory states of nodes progressively till all nodes are updated. In addition, for each node, the forgets gates are adaptively learned to capture different degrees of semantic correlation with neighboring nodes. Comprehensive evaluations on four diverse semantic object parsing datasets well demonstrate the significant superiority of our Graph LSTM over other state-of-the-art solutions.

18 pages

References in corpus (18)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Sequence to Sequence Learning with Neural Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Pixel Recurrent Neural Networks
• Recurrent Models of Visual Attention
• Fully Convolutional Networks for Semantic Segmentation
• Action Recognition using Visual Attention
• Deep Human Parsing with Active Template Regression
• Grid Long Short-Term Memory
• Fully Connected Deep Structured Networks
• Semantic Image Segmentation via Deep Parsing Network
• Attention to Scale: Scale-aware Semantic Image Segmentation
• Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture
• Semantic Object Parsing with Local-Global Long Short-Term Memory
• End-to-end people detection in crowded scenes
• Joint Object and Part Segmentation using Deep Learned Potentials
• Zoom Better to See Clearer: Human and Object Parsing with Hierarchical Auto-Zoom Net
• Superpixel Convolutional Networks using Bilateral Inceptions

Cited by in corpus (6)

• DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs
• RefineNet: Multi-Path Refinement Networks for High-Resolution Semantic Segmentation
• Joint Multi-Person Pose Estimation and Semantic Part Segmentation
• Scene Labeling using Gated Recurrent Units with Explicit Long Range Conditioning
• Object-Level Context Modeling For Scene Classification with Context-CNN
• Graphical RNN Models