Stand-Alone Self-Attention in Vision Models

Abstract

Convolutions are a fundamental building block of modern computer vision systems. Recent approaches have argued for going beyond convolutions in order to capture long-range dependencies. These efforts focus on augmenting convolutional models with content-based interactions, such as self-attention and non-local means, to achieve gains on a number of vision tasks. The natural question that arises is whether attention can be a stand-alone primitive for vision models instead of serving as just an augmentation on top of convolutions. In developing and testing a pure self-attention vision model, we verify that self-attention can indeed be an effective stand-alone layer. A simple procedure of replacing all instances of spatial convolutions with a form of self-attention applied to ResNet model produces a fully self-attentional model that outperforms the baseline on ImageNet classification with 12% fewer FLOPS and 29% fewer parameters. On COCO object detection, a pure self-attention model matches the mAP of a baseline RetinaNet while having 39% fewer FLOPS and 34% fewer parameters. Detailed ablation studies demonstrate that self-attention is especially impactful when used in later layers. These results establish that stand-alone self-attention is an important addition to the vision practitioner's toolbox.

References in corpus (13)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• DeepPose: Human Pose Estimation via Deep Neural Networks
• Convolutional Sequence to Sequence Learning
• Attention-Based Models for Speech Recognition
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Searching for Efficient Multi-Scale Architectures for Dense Image Prediction
• Pay Less Attention with Lightweight and Dynamic Convolutions
• Gauge Equivariant Convolutional Networks and the Icosahedral CNN
• An Empirical Study of Spatial Attention Mechanisms in Deep Networks
• Assessing the Scalability of Biologically-Motivated Deep Learning Algorithms and Architectures

Cited by in corpus (78)

• Attention Mechanisms in Computer Vision: A Survey
• CoAtNet: Marrying Convolution and Attention for All Data Sizes
• Visual Transformers: Token-based Image Representation and Processing for Computer Vision
• DeepViT: Towards Deeper Vision Transformer
• VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text
• Reformer: The Efficient Transformer
• Intriguing Properties of Vision Transformers
• Escaping the Big Data Paradigm with Compact Transformers
• NAS-FAS: Static-Dynamic Central Difference Network Search for Face Anti-Spoofing
• Focal Self-attention for Local-Global Interactions in Vision Transformers
• Revisiting ResNets: Improved Training and Scaling Strategies
• HRFormer: High-Resolution Transformer for Dense Prediction
• Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformer
• Training Vision Transformers for Image Retrieval
• The Origins and Prevalence of Texture Bias in Convolutional Neural Networks
• Do Vision Transformers See Like Convolutional Neural Networks?
• CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows
• SwinIR: Image Restoration Using Swin Transformer
• Multi-Head Attention: Collaborate Instead of Concatenate
• ConTNet: Why not use convolution and transformer at the same time?
• CMT: Convolutional Neural Networks Meet Vision Transformers
• Multiscale Vision Transformers
• An Image is Worth 16x16 Words, What is a Video Worth?
• Improving Face-Based Age Estimation with Attention-Based Dynamic Patch Fusion
• LambdaNetworks: Modeling Long-Range Interactions Without Attention
• MUSE: Parallel Multi-Scale Attention for Sequence to Sequence Learning
• An Attention Free Transformer
• Glance-and-Gaze Vision Transformer
• PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers
• Attentive Group Equivariant Convolutional Networks
• Joint Learning of Salient Object Detection, Depth Estimation and Contour Extraction
• Rethinking and Improving Relative Position Encoding for Vision Transformer
• Neural Language Generation: Formulation, Methods, and Evaluation
• Sequential Image-based Attention Network for Inferring Force Estimation without Haptic Sensor
• Reconstruction Student with Attention for Student-Teacher Pyramid Matching
• AutoFormer: Searching Transformers for Visual Recognition
• Spatially-Attentive Patch-Hierarchical Network for Adaptive Motion Deblurring
• Revisiting 3D ResNets for Video Recognition
• Fast Convergence of DETR with Spatially Modulated Co-Attention
• Convolution-enhanced Evolving Attention Networks
• CKConv: Continuous Kernel Convolution For Sequential Data
• Searching for Efficient Multi-Stage Vision Transformers
• Hierarchical Opacity Propagation for Image Matting
• Test-Time Personalization with a Transformer for Human Pose Estimation
• Hard-Attention for Scalable Image Classification
• Learning Fair Face Representation With Progressive Cross Transformer
• Representation Learning for Event-based Visuomotor Policies
• Multi-Scale Feature Aggregation by Cross-Scale Pixel-to-Region Relation Operation for Semantic Segmentation
• AGMB-Transformer: Anatomy-Guided Multi-Branch Transformer Network for Automated Evaluation of Root Canal Therapy
• Attention-based Image Upsampling
• Patch-Based Stochastic Attention for Image Editing
• Fibro-CoSANet: Pulmonary Fibrosis Prognosis Prediction using a Convolutional Self Attention Network
• Attention Aware Cost Volume Pyramid Based Multi-view Stereo Network for 3D Reconstruction
• Group Equivariant Stand-Alone Self-Attention For Vision
• Sampling Equivariant Self-attention Networks for Object Detection in Aerial Images
• Transformed CNNs: recasting pre-trained convolutional layers with self-attention
• Transformers as Strong Lens Detectors- From Simulation to Surveys
• Transformers are Deep Infinite-Dimensional Non-Mercer Binary Kernel Machines
• Affine Self Convolution
• Video-based Person Re-identification without Bells and Whistles
• H-Transformer-1D: Fast One-Dimensional Hierarchical Attention for Sequences
• VTAMIQ: Transformers for Attention Modulated Image Quality Assessment
• TBN-ViT: Temporal Bilateral Network with Vision Transformer for Video Scene Parsing
• DRF: A Framework for High-Accuracy Autonomous Driving Vehicle Modeling
• Token Pooling in Vision Transformers
• Domain Adaptation of NMT models for English-Hindi Machine Translation Task at AdapMT ICON 2020
• Probabilistic Attention for Interactive Segmentation
• Deeper or Wider Networks of Point Clouds with Self-attention?
• Bayesian Attention Modules
• 15 Keypoints Is All You Need
• Relational Mimic for Visual Adversarial Imitation Learning
• Knowledge as Invariance -- History and Perspectives of Knowledge-augmented Machine Learning
• Salient Object Detection Combining a Self-attention Module and a Feature Pyramid Network
• Cyclic orthogonal convolutions for long-range integration of features
• Gabor filter incorporated CNN for compression
• Adaptive Attention Span in Computer Vision
• CNN with large memory layers
• Interpretable UAV Collision Avoidance using Deep Reinforcement Learning