Hard-Attention for Scalable Image Classification

Abstract

Can we leverage high-resolution information without the unsustainable quadratic complexity to input scale? We propose Traversal Network (TNet), a novel multi-scale hard-attention architecture, which traverses image scale-space in a top-down fashion, visiting only the most informative image regions along the way. TNet offers an adjustable trade-off between accuracy and complexity, by changing the number of attended image locations. We compare our model against hard-attention baselines on ImageNet, achieving higher accuracy with less resources (FLOPs, processing time and memory). We further test our model on fMoW dataset, where we process satellite images of size up to $896\times 896$ px, getting up to $2.5$x faster processing compared to baselines operating on the same resolution, while achieving higher accuracy as well. TNet is modular, meaning that most classification models could be adopted as its backbone for feature extraction, making the reported performance gains orthogonal to benefits offered by existing optimized deep models. Finally, hard-attention guarantees a degree of interpretability to our model's predictions, without any extra cost beyond inference. Code is available at $\text{href}$.

References in corpus (23)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Rethinking Atrous Convolution for Semantic Image Segmentation
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• EfficientNetV2: Smaller Models and Faster Training
• Recurrent Models of Visual Attention
• Understanding the Effective Receptive Field in Deep Convolutional Neural Networks
• Multiple Object Recognition with Visual Attention
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Some Improvements on Deep Convolutional Neural Network Based Image Classification
• Multi-Scale Dense Networks for Resource Efficient Image Classification
• Stand-Alone Self-Attention in Vision Models
• Using Fast Weights to Attend to the Recent Past
• Attention for Fine-Grained Categorization
• Streaming convolutional neural networks for end-to-end learning with multi-megapixel images
• On Learning Where To Look
• Saccader: Improving Accuracy of Hard Attention Models for Vision
• Learning to Interpret Satellite Images in Global Scale Using Wikipedia
• Efficient Coarse-to-Fine Non-Local Module for the Detection of Small Objects

Cited by in corpus (2)

• Recurrent Attention Models with Object-centric Capsule Representation for Multi-object Recognition
• Efficient Classification of Very Large Images with Tiny Objects