Unsupervised learning of object semantic parts from internal states of CNNs by population encoding

Abstract

We address the key question of how object part representations can be found from the internal states of CNNs that are trained for high-level tasks, such as object classification. This work provides a new unsupervised method to learn semantic parts and gives new understanding of the internal representations of CNNs. Our technique is based on the hypothesis that semantic parts are represented by populations of neurons rather than by single filters. We propose a clustering technique to extract part representations, which we call Visual Concepts. We show that visual concepts are semantically coherent in that they represent semantic parts, and visually coherent in that corresponding image patches appear very similar. Also, visual concepts provide full spatial coverage of the parts of an object, rather than a few sparse parts as is typically found in keypoint annotations. Furthermore, We treat single visual concept as part detector and evaluate it for keypoint detection using the PASCAL3D+ dataset and for part detection using our newly annotated ImageNetPart dataset. The experiments demonstrate that visual concepts can be used to detect parts. We also show that some visual concepts respond to several semantic parts, provided these parts are visually similar. Thus visual concepts have the essential properties: semantic meaning and detection capability. Note that our ImageNetPart dataset gives rich part annotations which cover the whole object, making it useful for other part-related applications.

References in corpus (7)

• Intriguing properties of neural networks
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Understanding Neural Networks Through Deep Visualization
• Holistically-Nested Edge Detection
• Detect What You Can: Detecting and Representing Objects using Holistic Models and Body Parts
• Neural Activation Constellations: Unsupervised Part Model Discovery with Convolutional Networks
• Analyzing the Performance of Multilayer Neural Networks for Object Recognition

Cited by in corpus (16)

• What makes ImageNet good for transfer learning?
• Mitigating Adversarial Effects Through Randomization
• Adversarial Examples for Semantic Segmentation and Object Detection
• Towards falsifiable interpretability research
• TDAPNet: Prototype Network with Recurrent Top-Down Attention for Robust Object Classification under Partial Occlusion
• Visual Concepts and Compositional Voting
• Detecting Semantic Parts on Partially Occluded Objects
• Recurrent Feedback Improves Feedforward Representations in Deep Neural Networks
• Unsupervised Part Discovery from Contrastive Reconstruction
• Interpretable Neural Network Decoupling
• DeepVoting: A Robust and Explainable Deep Network for Semantic Part Detection under Partial Occlusion
• Class Subset Selection for Transfer Learning using Submodularity
• Explanations for Occluded Images
• Compositional Convolutional Neural Networks: A Deep Architecture with Innate Robustness to Partial Occlusion
• Deep Collaborative Learning for Visual Recognition
• Localizing Occluders with Compositional Convolutional Networks