Vision Transformer Pruning

Abstract

Vision transformer has achieved competitive performance on a variety of computer vision applications. However, their storage, run-time memory, and computational demands are hindering the deployment to mobile devices. Here we present a vision transformer pruning approach, which identifies the impacts of dimensions in each layer of transformer and then executes pruning accordingly. By encouraging dimension-wise sparsity in the transformer, important dimensions automatically emerge. A great number of dimensions with small importance scores can be discarded to achieve a high pruning ratio without significantly compromising accuracy. The pipeline for vision transformer pruning is as follows: 1) training with sparsity regularization; 2) pruning dimensions of linear projections; 3) fine-tuning. The reduced parameters and FLOPs ratios of the proposed algorithm are well evaluated and analyzed on ImageNet dataset to demonstrate the effectiveness of our proposed method.

arXiv commentAccepted by the KDD 2021 Workshop on Model Mining

References in corpus (13)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Distilling the Knowledge in a Neural Network
• mixup: Beyond Empirical Risk Minimization
• Deformable DETR: Deformable Transformers for End-to-End Object Detection
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Q8BERT: Quantized 8Bit BERT
• Convolutional neural networks with low-rank regularization
• Big Bird: Transformers for Longer Sequences
• End-to-End Video Instance Segmentation with Transformers
• ISTR: End-to-End Instance Segmentation with Transformers
• Searching for Low-Bit Weights in Quantized Neural Networks
• MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers
• Learning Low-Rank Approximation for CNNs

Cited by in corpus (4)

• A survey on efficient vision transformers: algorithms, techniques, and performance benchmarking
• Vision Transformers on the Edge: A Comprehensive Survey of Model Compression and Acceleration Strategies
• OnDev-LCT: On-Device Lightweight Convolutional Transformers towards federated learning
• Elastoformer: Enabling Dynamic Adaptivity via Elastic Model Transformation