Locally-Supervised Deep Hybrid Model for Scene Recognition

Abstract

Convolutional neural networks (CNN) have recently achieved remarkable successes in various image classification and understanding tasks. The deep features obtained at the top fully-connected layer of the CNN (FC-features) exhibit rich global semantic information and are extremely effective in image classification. On the other hand, the convolutional features in the middle layers of the CNN also contain meaningful local information, but are not fully explored for image representation. In this paper, we propose a novel Locally-Supervised Deep Hybrid Model (LS-DHM) that effectively enhances and explores the convolutional features for scene recognition. Firstly, we notice that the convolutional features capture local objects and fine structures of scene images, which yield important cues for discriminating ambiguous scenes, whereas these features are significantly eliminated in the highly-compressed FC representation. Secondly, we propose a new Local Convolutional Supervision (LCS) layer to enhance the local structure of the image by directly propagating the label information to the convolutional layers. Thirdly, we propose an efficient Fisher Convolutional Vector (FCV) that successfully rescues the orderless mid-level semantic information (e.g. objects and textures) of scene image. The FCV encodes the large-sized convolutional maps into a fixed-length mid-level representation, and is demonstrated to be strongly complementary to the high-level FC-features. Finally, both the FCV and FC-features are collaboratively employed in the LSDHM representation, which achieves outstanding performance in our experiments. It obtains 83.75% and 67.56% accuracies respectively on the heavily benchmarked MIT Indoor67 and SUN397 datasets, advancing the stat-of-the-art substantially.

To appear in IEEE Trans. on Image Processing, 2017

References in corpus (13)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• How transferable are features in deep neural networks?
• Action Recognition with Trajectory-Pooled Deep-Convolutional Descriptors
• Object Detectors Emerge in Deep Scene CNNs
• Text-Attentional Convolutional Neural Networks for Scene Text Detection
• Simultaneous Detection and Segmentation
• Hybrid CNN and Dictionary-Based Models for Scene Recognition and Domain Adaptation
• Training Deeper Convolutional Networks with Deep Supervision
• Knowledge Guided Disambiguation for Large-Scale Scene Classification with Multi-Resolution CNNs
• Places205-VGGNet Models for Scene Recognition
• Learning Contextual Dependencies with Convolutional Hierarchical Recurrent Neural Networks
• Deep Spatial Pyramid: The Devil is Once Again in the Details
• Local Color Contrastive Descriptor for Image Classification

Cited by in corpus (12)

• Knowledge Guided Disambiguation for Large-Scale Scene Classification with Multi-Resolution CNNs
• Weakly Supervised PatchNets: Describing and Aggregating Local Patches for Scene Recognition
• Deep Learning for Scene Classification: A Survey
• From Volcano to Toyshop: Adaptive Discriminative Region Discovery for Scene Recognition
• CurriculumNet: Weakly Supervised Learning from Large-Scale Web Images
• Indoor image representation by high-level semantic features
• Recent Advances in Scene Image Representation and Classification
• Tag-based Semantic Features for Scene Image Classification
• Sunrise or Sunset: Selective Comparison Learning for Subtle Attribute Recognition
• Object-Level Context Modeling For Scene Classification with Context-CNN
• Scene-based Factored Attention for Image Captioning
• Handcrafted vs Deep Learning Classification for Scalable Video QoE Modeling