Saccader: Improving Accuracy of Hard Attention Models for Vision

Abstract

Although deep convolutional neural networks achieve state-of-the-art performance across nearly all image classification tasks, their decisions are difficult to interpret. One approach that offers some level of interpretability by design is \textit{hard attention}, which uses only relevant portions of the image. However, training hard attention models with only class label supervision is challenging, and hard attention has proved difficult to scale to complex datasets. Here, we propose a novel hard attention model, which we term Saccader. Key to Saccader is a pretraining step that requires only class labels and provides initial attention locations for policy gradient optimization. Our best models narrow the gap to common ImageNet baselines, achieving $75\%$ top-1 and $91\%$ top-5 while attending to less than one-third of the image.

arXiv comment33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada

References in corpus (14)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Understanding Neural Networks Through Deep Visualization
• Explaining NonLinear Classification Decisions with Deep Taylor Decomposition
• Recurrent Models of Visual Attention
• DRAW: A Recurrent Neural Network For Image Generation
• Multiple Object Recognition with Visual Attention
• Visualizing Deep Neural Network Decisions: Prediction Difference Analysis
• Technical Report on the CleverHans v2.1.0 Adversarial Examples Library
• Adversarial Risk and the Dangers of Evaluating Against Weak Attacks
• The (Un)reliability of saliency methods
• Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet
• Foveation-based Mechanisms Alleviate Adversarial Examples

Cited by in corpus (6)

• Concept Whitening for Interpretable Image Recognition
• Reflective-Net: Learning from Explanations
• Hard-Attention for Scalable Image Classification
• Active Perception and Representation for Robotic Manipulation
• Recurrent Attention Models with Object-centric Capsule Representation for Multi-object Recognition
• Localizing Interpretable Multi-scale informative Patches Derived from Media Classification Task