Focal Self-attention for Local-Global Interactions in Vision Transformers

Abstract

Recently, Vision Transformer and its variants have shown great promise on various computer vision tasks. The ability of capturing short- and long-range visual dependencies through self-attention is arguably the main source for the success. But it also brings challenges due to quadratic computational overhead, especially for the high-resolution vision tasks (e.g., object detection). In this paper, we present focal self-attention, a new mechanism that incorporates both fine-grained local and coarse-grained global interactions. Using this new mechanism, each token attends the closest surrounding tokens at fine granularity but the tokens far away at coarse granularity, and thus can capture both short- and long-range visual dependencies efficiently and effectively. With focal self-attention, we propose a new variant of Vision Transformer models, called Focal Transformer, which achieves superior performance over the state-of-the-art vision Transformers on a range of public image classification and object detection benchmarks. In particular, our Focal Transformer models with a moderate size of 51.1M and a larger size of 89.8M achieve 83.5 and 83.8 Top-1 accuracy, respectively, on ImageNet classification at 224x224 resolution. Using Focal Transformers as the backbones, we obtain consistent and substantial improvements over the current state-of-the-art Swin Transformers for 6 different object detection methods trained with standard 1x and 3x schedules. Our largest Focal Transformer yields 58.7/58.9 box mAPs and 50.9/51.3 mask mAPs on COCO mini-val/test-dev, and 55.4 mIoU on ADE20K for semantic segmentation, creating new SoTA on three of the most challenging computer vision tasks.

References in corpus (24)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• A Survey on Visual Transformer
• Transformers in Vision: A Survey
• Language Models are Few-Shot Learners
• Longformer: The Long-Document Transformer
• Deformable DETR: Deformable Transformers for End-to-End Object Detection
• SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers
• Conditional Positional Encodings for Vision Transformers
• DeepViT: Towards Deeper Vision Transformer
• Big Bird: Transformers for Longer Sequences
• Stand-Alone Self-Attention in Vision Models
• CvT: Introducing Convolutions to Vision Transformers
• Probabilistic two-stage detection
• End-to-End Object Detection with Adaptive Clustering Transformer
• Sparse R-CNN: End-to-End Object Detection with Learnable Proposals
• LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference
• Efficient Self-supervised Vision Transformers for Representation Learning
• End-to-End Video Instance Segmentation with Transformers
• Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking
• Compressive Transformers for Long-Range Sequence Modelling
• Vision Transformers with Patch Diversification
• MultiGrain: a unified image embedding for classes and instances
• VOLO: Vision Outlooker for Visual Recognition
• Trear: Transformer-based RGB-D Egocentric Action Recognition

Cited by in corpus (15)

• Transformers in Vision: A Survey
• Florence: A New Foundation Model for Computer Vision
• Swin Transformer V2: Scaling Up Capacity and Resolution
• Uformer: A General U-Shaped Transformer for Image Restoration
• VST++: Efficient and Stronger Visual Saliency Transformer
• Scaling Spike-driven Transformer with Efficient Spike Firing Approximation Training
• A Survey of Visual Transformers
• S${}^2$-MLPv2: Improved Spatial-Shift MLP Architecture for Vision
• EViT: An Eagle Vision Transformer with Bi-Fovea Self-Attention
• UniNeXt: Exploring A Unified Architecture for Vision Recognition
• Shunted Self-Attention via Multi-Scale Token Aggregation
• Mitigation of Spatial Nonstationarity with Vision Transformers
• PoNet: Pooling Network for Efficient Token Mixing in Long Sequences
• NomMer: Nominate Synergistic Context in Vision Transformer for Visual Recognition
• Ripple Attention for Visual Perception with Sub-quadratic Complexity