Hybrid CNN and Dictionary-Based Models for Scene Recognition and Domain Adaptation

Abstract

Convolutional neural network (CNN) has achieved state-of-the-art performance in many different visual tasks. Learned from a large-scale training dataset, CNN features are much more discriminative and accurate than the hand-crafted features. Moreover, CNN features are also transferable among different domains. On the other hand, traditional dictionarybased features (such as BoW and SPM) contain much more local discriminative and structural information, which is implicitly embedded in the images. To further improve the performance, in this paper, we propose to combine CNN with dictionarybased models for scene recognition and visual domain adaptation. Specifically, based on the well-tuned CNN models (e.g., AlexNet and VGG Net), two dictionary-based representations are further constructed, namely mid-level local representation (MLR) and convolutional Fisher vector representation (CFV). In MLR, an efficient two-stage clustering method, i.e., weighted spatial and feature space spectral clustering on the parts of a single image followed by clustering all representative parts of all images, is used to generate a class-mixture or a classspecific part dictionary. After that, the part dictionary is used to operate with the multi-scale image inputs for generating midlevel representation. In CFV, a multi-scale and scale-proportional GMM training strategy is utilized to generate Fisher vectors based on the last convolutional layer of CNN. By integrating the complementary information of MLR, CFV and the CNN features of the fully connected layer, the state-of-the-art performance can be achieved on scene recognition and domain adaptation problems. An interested finding is that our proposed hybrid representation (from VGG net trained on ImageNet) is also complementary with GoogLeNet and/or VGG-11 (trained on Place205) greatly.

Accepted by TCSVT on Sep.2015

References in corpus (12)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs
• How transferable are features in deep neural networks?
• Fully Convolutional Networks for Semantic Segmentation
• Learning Transferable Features with Deep Adaptation Networks
• Deep Domain Confusion: Maximizing for Domain Invariance
• Places205-VGGNet Models for Scene Recognition
• DeepID-Net: multi-stage and deformable deep convolutional neural networks for object detection
• Fully Convolutional Neural Networks for Crowd Segmentation
• Deep convolutional filter banks for texture recognition and segmentation
• Fisher Kernel for Deep Neural Activations

Cited by in corpus (15)

• Knowledge Guided Disambiguation for Large-Scale Scene Classification with Multi-Resolution CNNs
• Semantic-Aware Scene Recognition
• Locally-Supervised Deep Hybrid Model for Scene Recognition
• Scene Recognition with Prototype-agnostic Scene Layout
• Adversarial adaptive 1-D convolutional neural networks for bearing fault diagnosis under varying working condition
• Deep Learning for Scene Classification: A Survey
• From Volcano to Toyshop: Adaptive Discriminative Region Discovery for Scene Recognition
• Two-stream Collaborative Learning with Spatial-Temporal Attention for Video Classification
• Semantic-embedded Similarity Prototype for Scene Recognition
• Sunrise or Sunset: Selective Comparison Learning for Subtle Attribute Recognition
• Dense Residual Network: Enhancing Global Dense Feature Flow for Character Recognition
• Collaborative Layer-wise Discriminative Learning in Deep Neural Networks
• Superpixel based Class-Semantic Texton Occurrences for Natural Roadside Vegetation Segmentation
• Road Segmentation with Image-LiDAR Data Fusion
• FOSNet: An End-to-End Trainable Deep Neural Network for Scene Recognition