Learning Contextual Dependencies with Convolutional Hierarchical Recurrent Neural Networks

Abstract

Existing deep convolutional neural networks (CNNs) have shown their great success on image classification. CNNs mainly consist of convolutional and pooling layers, both of which are performed on local image areas without considering the dependencies among different image regions. However, such dependencies are very important for generating explicit image representation. In contrast, recurrent neural networks (RNNs) are well known for their ability of encoding contextual information among sequential data, and they only require a limited number of network parameters. General RNNs can hardly be directly applied on non-sequential data. Thus, we proposed the hierarchical RNNs (HRNNs). In HRNNs, each RNN layer focuses on modeling spatial dependencies among image regions from the same scale but different locations. While the cross RNN scale connections target on modeling scale dependencies among regions from the same location but different scales. Specifically, we propose two recurrent neural network models: 1) hierarchical simple recurrent network (HSRN), which is fast and has low computational cost; and 2) hierarchical long-short term memory recurrent network (HLSTM), which performs better than HSRN with the price of more computational cost. In this manuscript, we integrate CNNs with HRNNs, and develop end-to-end convolutional hierarchical recurrent neural networks (C-HRNNs). C-HRNNs not only make use of the representation power of CNNs, but also efficiently encodes spatial and scale dependencies among different image regions. On four of the most challenging object/scene image classification benchmarks, our C-HRNNs achieve state-of-the-art results on Places 205, SUN 397, MIT indoor, and competitive results on ILSVRC 2012.

References in corpus (12)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Going Deeper with Convolutions
• Recurrent Models of Visual Attention
• DRAW: A Recurrent Neural Network For Image Generation
• Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
• Modeling Temporal Dependencies in High-Dimensional Sequences: Application to Polyphonic Music Generation and Transcription
• Beyond Short Snippets: Deep Networks for Video Classification
• ReNet: A Recurrent Neural Network Based Alternative to Convolutional Networks
• Learning a Recurrent Visual Representation for Image Caption Generation
• Deep Visual-Semantic Alignments for Generating Image Descriptions
• DeepID-Net: multi-stage and deformable deep convolutional neural networks for object detection

Cited by in corpus (16)

• End-to-End CNN+LSTM Deep Learning Approach for Bearing Fault Diagnosis
• Knowledge Guided Disambiguation for Large-Scale Scene Classification with Multi-Resolution CNNs
• Recent Advances in Object Detection in the Age of Deep Convolutional Neural Networks
• Locally-Supervised Deep Hybrid Model for Scene Recognition
• Deep Learning Algorithms with Applications to Video Analytics for A Smart City: A Survey
• Weakly Supervised PatchNets: Describing and Aggregating Local Patches for Scene Recognition
• Scene Recognition with Prototype-agnostic Scene Layout
• Vehicle-Rear: A New Dataset to Explore Feature Fusion for Vehicle Identification Using Convolutional Neural Networks
• Deep Learning for Scene Classification: A Survey
• Multi-level Contextual RNNs with Attention Model for Scene Labeling
• SANet: Structure-Aware Network for Visual Tracking
• Dense Recurrent Neural Networks for Scene Labeling
• Leveraging Self-Supervised Learning for Scene Classification in Child Sexual Abuse Imagery
• Cosheaf Theoretical Constructions in Networks and Persistent Homology
• Hierarchical Spatial Sum-Product Networks for Action Recognition in Still Images
• Scene Parsing via Dense Recurrent Neural Networks with Attentional Selection