Glance and Focus: a Dynamic Approach to Reducing Spatial Redundancy in Image Classification

Abstract

The accuracy of deep convolutional neural networks (CNNs) generally improves when fueled with high resolution images. However, this often comes at a high computational cost and high memory footprint. Inspired by the fact that not all regions in an image are task-relevant, we propose a novel framework that performs efficient image classification by processing a sequence of relatively small inputs, which are strategically selected from the original image with reinforcement learning. Such a dynamic decision process naturally facilitates adaptive inference at test time, i.e., it can be terminated once the model is sufficiently confident about its prediction and thus avoids further redundant computation. Notably, our framework is general and flexible as it is compatible with most of the state-of-the-art light-weighted CNNs (such as MobileNets, EfficientNets and RegNets), which can be conveniently deployed as the backbone feature extractor. Experiments on ImageNet show that our method consistently improves the computational efficiency of a wide variety of deep models. For example, it further reduces the average latency of the highly efficient MobileNet-V3 on an iPhone XS Max by 20% without sacrificing accuracy. Code and pre-trained models are available at https://github.com/blackfeather-wang/GFNet-Pytorch.

arXiv commentAccepted by NeurIPS 2020

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• End to End Learning for Self-Driving Cars
• MobileNetV2: Inverted Residuals and Linear Bottlenecks
• Recurrent Models of Visual Attention
• Rethinking the Value of Network Pruning
• Channel Gating Neural Networks
• Spatially Adaptive Inference with Stochastic Feature Sampling and Interpolation

Cited by in corpus (5)

• DPT: Deformable Patch-based Transformer for Visual Recognition
• Intelligent machines work in unstructured environments by differential neuromorphic computing
• IA-RED${}^2$: Interpretability-Aware Redundancy Reduction for Vision Transformers
• Collaboration of Experts: Achieving 80% Top-1 Accuracy on ImageNet with 100M FLOPs
• Smart Scissor: Coupling Spatial Redundancy Reduction and CNN Compression for Embedded Hardware