Localizing Interpretable Multi-scale informative Patches Derived from Media Classification Task

Abstract

Deep convolutional neural networks (CNN) always depend on wider receptive field (RF) and more complex non-linearity to achieve state-of-the-art performance, while suffering the increased difficult to interpret how relevant patches contribute the final prediction. In this paper, we construct an interpretable AnchorNet equipped with our carefully designed RFs and linearly spatial aggregation to provide patch-wise interpretability of the input media meanwhile localizing multi-scale informative patches only supervised on media-level labels without any extra bounding box annotations. Visualization of localized informative image and text patches show the superior multi-scale localization capability of AnchorNet. We further use localized patches for downstream classification tasks across widely applied networks. Experimental results demonstrate that replacing the original inputs with their patches for classification can get a clear inference acceleration with only tiny performance degradation, which proves that localized patches can indeed retain the most semantics and evidences of the original inputs.

References in corpus (8)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• CNN: Single-label to Multi-label
• SmoothGrad: removing noise by adding noise
• Multi-Scale Dense Networks for Resource Efficient Image Classification
• MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning
• Deep FisherNet for Object Classification
• Saccader: Improving Accuracy of Hard Attention Models for Vision
• Gated Convolutional Networks with Hybrid Connectivity for Image Classification