Patch Slimming for Efficient Vision Transformers

Abstract

This paper studies the efficiency problem for visual transformers by excavating redundant calculation in given networks. The recent transformer architecture has demonstrated its effectiveness for achieving excellent performance on a series of computer vision tasks. However, similar to that of convolutional neural networks, the huge computational cost of vision transformers is still a severe issue. Considering that the attention mechanism aggregates different patches layer-by-layer, we present a novel patch slimming approach that discards useless patches in a top-down paradigm. We first identify the effective patches in the last layer and then use them to guide the patch selection process of previous layers. For each layer, the impact of a patch on the final output feature is approximated and patches with less impact will be removed. Experimental results on benchmark datasets demonstrate that the proposed method can significantly reduce the computational costs of vision transformers without affecting their performances. For example, over 45% FLOPs of the ViT-Ti model can be reduced with only 0.2% top-1 accuracy drop on the ImageNet dataset.

arXiv commentThis paper is accepted by CVPR 2022

References in corpus (12)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Distilling the Knowledge in a Neural Network
• Language Models are Few-Shot Learners
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Pruning Filters for Efficient ConvNets
• Learning Structured Sparsity in Deep Neural Networks
• Knowledge Distillation by On-the-Fly Native Ensemble
• Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers
• Pre-Trained Image Processing Transformer
• Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
• Training Binary Neural Networks through Learning with Noisy Supervision
• PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination

Cited by in corpus (6)

• A Survey on Visual Transformer
• Chasing Sparsity in Vision Transformers: An End-to-End Exploration
• IA-RED${}^2$: Interpretability-Aware Redundancy Reduction for Vision Transformers
• Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition
• A Unified Pruning Framework for Vision Transformers
• Post-Training Quantization for Vision Transformer