Armour: Generalizable Compact Self-Attention for Vision Transformers

Abstract

Attention-based transformer networks have demonstrated promising potential as their applications extend from natural language processing to vision. However, despite the recent improvements, such as sub-quadratic attention approximation and various training enhancements, the compact vision transformers to date using the regular attention still fall short in comparison with its convnet counterparts, in terms of \textit{accuracy,} \textit{model size}, \textit{and} \textit{throughput}. This paper introduces a compact self-attention mechanism that is fundamental and highly generalizable. The proposed method reduces redundancy and improves efficiency on top of the existing attention optimizations. We show its drop-in applicability for both the regular attention mechanism and some most recent variants in vision transformers. As a result, we produced smaller and faster models with the same or better accuracies.

References in corpus (12)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Linformer: Self-Attention with Linear Complexity
• Generating Long Sequences with Sparse Transformers
• SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning
• Reformer: The Efficient Transformer
• CvT: Introducing Convolutions to Vision Transformers
• Pre-Trained Image Processing Transformer
• Rethinking Attention with Performers
• LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference
• End-to-End Dense Video Captioning with Masked Transformer
• Learning Joint Spatial-Temporal Transformations for Video Inpainting
• Rethinking Transformer-based Set Prediction for Object Detection