Visual Transformers: Token-based Image Representation and Processing for Computer Vision

Abstract

Computer vision has achieved remarkable success by (a) representing images as uniformly-arranged pixel arrays and (b) convolving highly-localized features. However, convolutions treat all image pixels equally regardless of importance; explicitly model all concepts across all images, regardless of content; and struggle to relate spatially-distant concepts. In this work, we challenge this paradigm by (a) representing images as semantic visual tokens and (b) running transformers to densely model token relationships. Critically, our Visual Transformer operates in a semantic token space, judiciously attending to different image parts based on context. This is in sharp contrast to pixel-space transformers that require orders-of-magnitude more compute. Using an advanced training recipe, our VTs significantly outperform their convolutional counterparts, raising ResNet accuracy on ImageNet top-1 by 4.6 to 7 points while using fewer FLOPs and parameters. For semantic segmentation on LIP and COCO-stuff, VT-based feature pyramid networks (FPN) achieve 0.35 points higher mIoU while reducing the FPN module's FLOPs by 6.5x.

References in corpus (12)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Gather-Excite: Exploiting Feature Context in Convolutional Neural Networks
• Stand-Alone Self-Attention in Vision Models
• LatentGNN: Learning Efficient Non-local Relations for Visual Recognition
• Exploring Self-attention for Image Recognition
• FBNetV2: Differentiable Neural Architecture Search for Spatial and Channel Dimensions
• FBNetV3: Joint Architecture-Recipe Search using Predictor Pretraining

Cited by in corpus (53)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• A Survey on Visual Transformer
• ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases
• A novel time-frequency Transformer based on self-attention mechanism and its application in fault diagnosis of rolling bearings
• TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up
• Deep Learning-based 3D Point Cloud Classification: A Systematic Survey and Outlook
• TriTransNet: RGB-D Salient Object Detection with a Triplet Transformer Embedding Network
• CenterCLIP: Token Clustering for Efficient Text-Video Retrieval
• Perceiver: General Perception with Iterative Attention
• Pre-Trained Image Processing Transformer
• Do Vision Transformers See Like Convolutional Neural Networks?
• Sparse R-CNN: End-to-End Object Detection with Learnable Proposals
• LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference
• SwinIR: Image Restoration Using Swin Transformer
• Multiscale Vision Transformers
• Line as a Visual Sentence: Context-aware Line Descriptor for Visual Localization
• Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers
• A Survey of Visual Transformers
• Prediction Calibration for Generalized Few-shot Semantic Segmentation
• Evolving Modular Soft Robots without Explicit Inter-Module Communication using Local Self-Attention
• STB-VMM: Swin Transformer Based Video Motion Magnification
• Local-to-Global Self-Attention in Vision Transformers
• Homogeneous Tokenizer Matters: Homogeneous Visual Tokenizer for Remote Sensing Image Understanding
• Visual Parser: Representing Part-whole Hierarchies with Transformers
• Mask-guided Spectral-wise Transformer for Efficient Hyperspectral Image Reconstruction
• Simpler is Better: Few-shot Semantic Segmentation with Classifier Weight Transformer
• Analogous to Evolutionary Algorithm: Designing a Unified Sequence Model
• TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?
• Distinguishing a planetary transit from false positives: a Transformer-based classification for planetary transit signals
• Variable-rate discrete representation learning
• gSwin: Gated MLP Vision Model with Hierarchical Structure of Shifted Window
• Visual Keyword Spotting with Attention
• GroupFormer: Group Activity Recognition with Clustered Spatial-Temporal Transformer
• An Image Patch is a Wave: Phase-Aware Vision MLP
• Visual Transformer for Task-aware Active Learning
• Dual Contrastive Loss and Attention for GANs
• Bridging Global Context Interactions for High-Fidelity Image Completion
• Subdimensional Expansion Using Attention-Based Learning For Multi-Agent Path Finding
• HR-NAS: Searching Efficient High-Resolution Neural Architectures with Lightweight Transformers
• Transformed CNNs: recasting pre-trained convolutional layers with self-attention
• Sub-word Level Lip Reading With Visual Attention
• Aggregating Soft Labels from Crowd Annotations Improves Uncertainty Estimation Under Distribution Shift
• Co-advise: Cross Inductive Bias Distillation
• TS-CAM: Token Semantic Coupled Attention Map for Weakly Supervised Object Localization
• CT-block: a novel local and global features extractor for point cloud
• Transformer Assisted Convolutional Network for Cell Instance Segmentation
• OadTR: Online Action Detection with Transformers
• Token Pooling in Vision Transformers
• Pathologies in priors and inference for Bayesian transformers
• Vision Pair Learning: An Efficient Training Framework for Image Classification
• Synthesizing Photorealistic Images with Deep Generative Learning
• The Brownian motion in the transformer model
• T-EMDE: Sketching-based global similarity for cross-modal retrieval