Knowledge Guided Disambiguation for Large-Scale Scene Classification with Multi-Resolution CNNs

Abstract

Convolutional Neural Networks (CNNs) have made remarkable progress on scene recognition, partially due to these recent large-scale scene datasets, such as the Places and Places2. Scene categories are often defined by multi-level information, including local objects, global layout, and background environment, thus leading to large intra-class variations. In addition, with the increasing number of scene categories, label ambiguity has become another crucial issue in large-scale classification. This paper focuses on large-scale scene recognition and makes two major contributions to tackle these issues. First, we propose a multi-resolution CNN architecture that captures visual content and structure at multiple levels. The multi-resolution CNNs are composed of coarse resolution CNNs and fine resolution CNNs, which are complementary to each other. Second, we design two knowledge guided disambiguation techniques to deal with the problem of label ambiguity. (i) We exploit the knowledge from the confusion matrix computed on validation data to merge ambiguous classes into a super category. (ii) We utilize the knowledge of extra networks to produce a soft label for each image. Then the super categories or soft labels are employed to guide CNN training on the Places2. We conduct extensive experiments on three large-scale image datasets (ImageNet, Places, and Places2), demonstrating the effectiveness of our approach. Furthermore, our method takes part in two major scene recognition challenges, and achieves the second place at the Places2 challenge in ILSVRC 2015, and the first place at the LSUN challenge in CVPR 2016. Finally, we directly test the learned representations on other scene benchmarks, and obtain the new state-of-the-art results on the MIT Indoor67 (86.7\%) and SUN397 (72.0\%). We release the code and models at~\url{https://github.com/wanglimin/MRCNN-Scene-Recognition}.

arXiv commentTo appear in IEEE Transactions on Image Processing. Code and models are available at https://github.com/wanglimin/MRCNN-Scene-Recognition

References in corpus (17)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Distilling the Knowledge in a Neural Network
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
• FitNets: Hints for Thin Deep Nets
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• Weakly Supervised Fine-Grained Image Categorization
• Places: An Image Database for Deep Scene Understanding
• Hybrid CNN and Dictionary-Based Models for Scene Recognition and Domain Adaptation
• Training Deeper Convolutional Networks with Deep Supervision
• Scene recognition with CNNs: objects, scales and dataset bias
• Places205-VGGNet Models for Scene Recognition
• Learning Contextual Dependencies with Convolutional Hierarchical Recurrent Neural Networks
• Locally-Supervised Deep Hybrid Model for Scene Recognition
• Deep Spatial Pyramid: The Devil is Once Again in the Details
• Transferring Object-Scene Convolutional Neural Networks for Event Recognition in Still Images

Cited by in corpus (9)

• Locally-Supervised Deep Hybrid Model for Scene Recognition
• Scene Recognition with Prototype-agnostic Scene Layout
• From Volcano to Toyshop: Adaptive Discriminative Region Discovery for Scene Recognition
• Deep Learning for Scene Classification: A Survey
• SRRM: Semantic Region Relation Model for Indoor Scene Recognition
• Translate-to-Recognize Networks for RGB-D Scene Recognition
• Semantic-guided modeling of spatial relation and object co-occurrence for indoor scene recognition
• Knowledge Integration Networks for Action Recognition
• FOSNet: An End-to-End Trainable Deep Neural Network for Scene Recognition