Towards Accurate Post-Training Quantization for Vision Transformer

Abstract

Vision transformer emerges as a potential architecture for vision tasks. However, the intense computation and non-negligible delay hinder its application in the real world. As a widespread model compression technique, existing post-training quantization methods still cause severe performance drops. We find the main reasons lie in (1) the existing calibration metric is inaccurate in measuring the quantization influence for extremely low-bit representation, and (2) the existing quantization paradigm is unfriendly to the power-law distribution of Softmax. Based on these observations, we propose a novel Accurate Post-training Quantization framework for Vision Transformer, namely APQ-ViT. We first present a unified Bottom-elimination Blockwise Calibration scheme to optimize the calibration metric to perceive the overall quantization disturbance in a blockwise manner and prioritize the crucial quantization errors that influence more on the final output. Then, we design a Matthew-effect Preserving Quantization for Softmax to maintain the power-law character and keep the function of the attention mechanism. Comprehensive experiments on large-scale classification and detection datasets demonstrate that our APQ-ViT surpasses the existing post-training quantization methods by convincing margins, especially in lower bit-width settings (e.g., averagely up to 5.17% improvement for classification and 24.43% for detection on W4A4). We also highlight that APQ-ViT enjoys versatility and works well on diverse transformer variants.

arXiv comment9 pages, 5 figures, accepted by ACM Multimedia 2022

References in corpus (10)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Longformer: The Long-Document Transformer
• meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting
• Attention-Based Transformers for Instance Segmentation of Cells in Microstructures
• QuadTree Attention for Vision Transformers
• EasyQuant: Post-training Quantization via Scale Optimization
• Towards Efficient Post-training Quantization of Pre-trained Language Models
• Structured Binary Neural Networks for Accurate Image Classification and Semantic Segmentation
• Memorized Sparse Backpropagation

Cited by in corpus (7)

• A survey on efficient vision transformers: algorithms, techniques, and performance benchmarking
• LLM-FP4: 4-Bit Floating-Point Quantized Transformers
• Vision Transformers on the Edge: A Comprehensive Survey of Model Compression and Acceleration Strategies
• Efficient Multimodal Large Language Models: A Survey
• LGViT: Dynamic Early Exiting for Accelerating Vision Transformer
• OnDev-LCT: On-Device Lightweight Convolutional Transformers towards federated learning
• TransAxx: Efficient Transformers with Approximate Computing