Visual Concepts and Compositional Voting

Abstract

It is very attractive to formulate vision in terms of pattern theory \cite{Mumford2010pattern}, where patterns are defined hierarchically by compositions of elementary building blocks. But applying pattern theory to real world images is currently less successful than discriminative methods such as deep networks. Deep networks, however, are black-boxes which are hard to interpret and can easily be fooled by adding occluding objects. It is natural to wonder whether by better understanding deep networks we can extract building blocks which can be used to develop pattern theoretic models. This motivates us to study the internal representations of a deep network using vehicle images from the PASCAL3D+ dataset. We use clustering algorithms to study the population activities of the features and extract a set of visual concepts which we show are visually tight and correspond to semantic parts of vehicles. To analyze this we annotate these vehicles by their semantic parts to create a new dataset, VehicleSemanticParts, and evaluate visual concepts as unsupervised part detectors. We show that visual concepts perform fairly well but are outperformed by supervised discriminative methods such as Support Vector Machines (SVM). We next give a more detailed analysis of visual concepts and how they relate to semantic parts. Following this, we use the visual concepts as building blocks for a simple pattern theoretical model, which we call compositional voting. In this model several visual concepts combine to detect semantic parts. We show that this approach is significantly better than discriminative methods like SVM and deep networks trained specifically for semantic part detection. Finally, we return to studying occlusion by creating an annotated dataset with occlusion, called VehicleOcclusion, and show that compositional voting outperforms even deep networks when the amount of occlusion becomes large.

It is accepted by Annals of Mathematical Sciences and Applications

References in corpus (7)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Detect What You Can: Detecting and Representing Objects using Holistic Models and Body Parts
• Neural Activation Constellations: Unsupervised Part Model Discovery with Convolutional Networks
• von Mises-Fisher Mixture Model-based Deep learning: Application to Face Verification
• Unsupervised learning of object semantic parts from internal states of CNNs by population encoding
• Amodal Instance Segmentation
• Amodal Completion and Size Constancy in Natural Scenes

Cited by in corpus (8)

• Shortcut Learning in Deep Neural Networks
• Mitigating Adversarial Effects Through Randomization
• TDAPNet: Prototype Network with Recurrent Top-Down Attention for Robust Object Classification under Partial Occlusion
• Robust neural circuit reconstruction from serial electron microscopy with convolutional recurrent networks
• BUZz: BUffer Zones for defending adversarial examples in image classification
• Compositional Convolutional Neural Networks: A Deep Architecture with Innate Robustness to Partial Occlusion
• Livewired Neural Networks: Making Neurons That Fire Together Wire Together
• Unsupervised Part Discovery via Feature Alignment