Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition

Abstract

Vision Transformers (ViT) have achieved remarkable success in large-scale image recognition. They split every 2D image into a fixed number of patches, each of which is treated as a token. Generally, representing an image with more tokens would lead to higher prediction accuracy, while it also results in drastically increased computational cost. To achieve a decent trade-off between accuracy and speed, the number of tokens is empirically set to 16x16 or 14x14. In this paper, we argue that every image has its own characteristics, and ideally the token number should be conditioned on each individual input. In fact, we have observed that there exist a considerable number of "easy" images which can be accurately predicted with a mere number of 4x4 tokens, while only a small fraction of "hard" ones need a finer representation. Inspired by this phenomenon, we propose a Dynamic Transformer to automatically configure a proper number of tokens for each input image. This is achieved by cascading multiple Transformers with increasing numbers of tokens, which are sequentially activated in an adaptive fashion at test time, i.e., the inference is terminated once a sufficiently confident prediction is produced. We further design efficient feature reuse and relationship reuse mechanisms across different components of the Dynamic Transformer to reduce redundant computations. Extensive empirical results on ImageNet, CIFAR-10, and CIFAR-100 demonstrate that our method significantly outperforms the competitive baselines in terms of both theoretical computational efficiency and practical inference speed. Code and pre-trained models (based on PyTorch and MindSpore) are available at https://github.com/blackfeather-wang/Dynamic-Vision-Transformer and https://github.com/blackfeather-wang/Dynamic-Vision-Transformer-MindSpore.

arXiv commentAccepted by NeurIPS 2021

References in corpus (17)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Transformer in Transformer
• Conditional Positional Encodings for Vision Transformers
• DeepViT: Towards Deeper Vision Transformer
• DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification
• LocalViT: Analyzing Locality in Vision Transformers
• Multi-Scale Dense Networks for Resource Efficient Image Classification
• Dynamic Convolutions: Exploiting Spatial Sparsity for Faster Inference
• CvT: Introducing Convolutions to Vision Transformers
• Training Vision Transformers for Image Retrieval
• LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference
• TrTr: Visual Tracking with Transformer
• Multiscale Vision Transformers
• Dynamic Neural Networks: A Survey
• MOTR: End-to-End Multiple-Object Tracking with Transformer
• Patch Slimming for Efficient Vision Transformers
• Adaptive Focus for Efficient Video Recognition