Transformers in Vision: A Survey

Abstract

Astounding results from Transformer models on natural language tasks have intrigued the vision community to study their application to computer vision problems. Among their salient benefits, Transformers enable modeling long dependencies between input sequence elements and support parallel processing of sequence as compared to recurrent networks e.g., Long short-term memory (LSTM). Different from convolutional networks, Transformers require minimal inductive biases for their design and are naturally suited as set-functions. Furthermore, the straightforward design of Transformers allows processing multiple modalities (e.g., images, videos, text and speech) using similar processing blocks and demonstrates excellent scalability to very large capacity networks and huge datasets. These strengths have led to exciting progress on a number of vision tasks using Transformer networks. This survey aims to provide a comprehensive overview of the Transformer models in the computer vision discipline. We start with an introduction to fundamental concepts behind the success of Transformers i.e., self-attention, large-scale pre-training, and bidirectional encoding. We then cover extensive applications of transformers in vision including popular recognition tasks (e.g., image classification, object detection, action recognition, and segmentation), generative modeling, multi-modal tasks (e.g., visual-question answering, visual reasoning, and visual grounding), video processing (e.g., activity recognition, video forecasting), low-level vision (e.g., image super-resolution, image enhancement, and colorization) and 3D analysis (e.g., point cloud classification and segmentation). We compare the respective advantages and limitations of popular techniques both in terms of architectural design and their experimental value. Finally, we provide an analysis on open research directions and possible future works.

arXiv comment30 pages (Accepted in ACM Computing Surveys December 2021)

References in corpus (103)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Distilling the Knowledge in a Neural Network
• Semi-Supervised Classification with Graph Convolutional Networks
• A Simple Framework for Contrastive Learning of Visual Representations
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Learning Transferable Visual Models From Natural Language Supervision
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Bootstrap your own latent: A new approach to self-supervised Learning
• Language Models are Few-Shot Learners
• The Kinetics Human Action Video Dataset
• PVT v2: Improved Baselines with Pyramid Vision Transformer
• Generative Adversarial Text to Image Synthesis
• Longformer: The Long-Document Transformer
• Improved Baselines with Momentum Contrastive Learning
• Conditional Image Generation with PixelCNN Decoders
• Deformable DETR: Deformable Transformers for End-to-End Object Detection
• NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding
• Is Space-Time Attention All You Need for Video Understanding?
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• VQA: Visual Question Answering
• Transformer in Transformer
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Linformer: Self-Attention with Linear Complexity
• VL-BERT: Pre-training of Generic Visual-Linguistic Representations
• CoAtNet: Marrying Convolution and Attention for All Data Sizes
• Twins: Revisiting the Design of Spatial Attention in Vision Transformers
• Efficient Neural Architecture Search via Parameter Sharing
• Contrastive Multiview Coding
• Generating Long Sequences with Sparse Transformers
• Conditional Positional Encodings for Vision Transformers
• Skeleton-based Action Recognition via Spatial and Temporal Transformer Networks
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• Axial Attention in Multidimensional Transformers
• A Short Note on the Kinetics-700-2020 Human Action Dataset
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• DeepViT: Towards Deeper Vision Transformer
• Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization
• Reformer: The Efficient Transformer
• Intriguing Properties of Vision Transformers
• Escaping the Big Data Paradigm with Compact Transformers
• LocalViT: Analyzing Locality in Vision Transformers
• Focal Self-attention for Local-Global Interactions in Vision Transformers
• Spatial Temporal Transformer Network for Skeleton-based Action Recognition
• MARTA GANs: Unsupervised Representation Learning for Remote Sensing Image Classification
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• XCiT: Cross-Covariance Image Transformers
• Learning Video Representations using Contrastive Bidirectional Transformer
• HAT: Hardware-Aware Transformers for Efficient Natural Language Processing
• Towards Automatic Learning of Procedures from Web Instructional Videos
• Perceiver IO: A General Architecture for Structured Inputs & Outputs
• Synthesizer: Rethinking Self-Attention in Transformer Models
• Restormer: Efficient Transformer for High-Resolution Image Restoration
• You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detection
• Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey
• CvT: Introducing Convolutions to Vision Transformers
• ResT: An Efficient Transformer for Visual Recognition
• All Tokens Matter: Token Labeling for Training Better Vision Transformers
• Towards a Neural Statistician
• Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction
• Pix2seq: A Language Modeling Framework for Object Detection
• Perceiver: General Perception with Iterative Attention
• Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformer
• Pre-Trained Image Processing Transformer
• Random Feature Attention
• Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
• Self-Supervised Learning with Swin Transformers
• Uformer: A General U-Shaped Transformer for Image Restoration
• Sharpness-Aware Minimization for Efficiently Improving Generalization
• CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows
• When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations
• Attention-Based Transformers for Instance Segmentation of Cells in Microstructures
• TransReID: Transformer-based Object Re-Identification
• RegionViT: Regional-to-Local Attention for Vision Transformers
• LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference
• Efficient Self-supervised Vision Transformers for Representation Learning
• Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet
• Cross-Domain Transferability of Adversarial Perturbations
• Sparse Sinkhorn Attention
• COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learning
• Referring Transformer: A One-step Approach to Multi-task Visual Grounding
• Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
• Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation
• End-to-End Video Instance Segmentation with Transformers
• On the Turing Completeness of Modern Neural Network Architectures
• CrossTransformers: spatially-aware few-shot transfer
• Multiscale Vision Transformers
• A Universal Representation Transformer Layer for Few-Shot Image Classification
• LambdaNetworks: Modeling Long-Range Interactions Without Attention
• End-to-End Human Pose and Mesh Reconstruction with Transformers
• End-to-End Dense Video Captioning with Masked Transformer
• Parameter Efficient Multimodal Transformers for Video Representation Learning
• AutoFormer: Searching Transformers for Visual Recognition
• SceneFormer: Indoor Scene Generation with Transformers
• Deep Amortized Clustering
• Topological Planning with Transformers for Vision-and-Language Navigation
• On Improving Adversarial Transferability of Vision Transformers
• Attention, please! A survey of Neural Attention Models in Deep Learning
• Multi-Scale Vision Longformer: A New Vision Transformer for High-Resolution Image Encoding
• GLiT: Neural Architecture Search for Global and Local Image Transformer
• Long-Short Temporal Contrastive Learning of Video Transformers
• Grounded Video Description

Cited by in corpus (91)

• Attention Mechanisms in Computer Vision: A Survey
• Deep Neural Networks and Tabular Data: A Survey
• CoAtNet: Marrying Convolution and Attention for All Data Sizes
• A Survey of Human-in-the-loop for Machine Learning
• Human Action Recognition from Various Data Modalities: A Review
• Deep Learning for Time Series Anomaly Detection: A Survey
• Multimodal Fusion Transformer for Remote Sensing Image Classification
• Intriguing Properties of Vision Transformers
• Combining EfficientNet and Vision Transformers for Video Deepfake Detection
• Focal Self-attention for Local-Global Interactions in Vision Transformers
• Building extraction with vision transformer
• Vision Transformers For Weeds and Crops Classification Of High Resolution UAV Images
• Video Transformers: A Survey
• Short and Long Range Relation Based Spatio-Temporal Transformer for Micro-Expression Recognition
• Looking Outside the Window: Wide-Context Transformer for the Semantic Segmentation of High-Resolution Remote Sensing Images
• A Practical Survey on Faster and Lighter Transformers
• How to avoid machine learning pitfalls: a guide for academic researchers
• Query2Label: A Simple Transformer Way to Multi-Label Classification
• Self-Supervised Masked Convolutional Transformer Block for Anomaly Detection
• Deep Contextual Video Compression
• Backbones-Review: Feature Extraction Networks for Deep Learning and Deep Reinforcement Learning Approaches
• CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation
• Is it Time to Replace CNNs with Transformers for Medical Images?
• Automated Detection and Forecasting of COVID-19 using Deep Learning Techniques: A Review
• TransReID: Transformer-based Object Re-Identification
• VTGAN: Semi-supervised Retinal Image Synthesis and Disease Prediction using Vision Transformers
• A review on vision-based analysis for automatic dietary assessment
• Self-Contrastive Learning with Hard Negative Sampling for Self-supervised Point Cloud Learning
• Beyond Self-attention: External Attention using Two Linear Layers for Visual Tasks
• A review of Generative Adversarial Networks for Electronic Health Records: applications, evaluation measures and data sources
• Swin-transformer-yolov5 For Real-time Wine Grape Bunch Detection
• Multi-Stage Progressive Image Restoration
• ISTR: End-to-End Instance Segmentation with Transformers
• Patch Similarity Aware Data-Free Quantization for Vision Transformers
• ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis
• Neuromorphic Camera Denoising using Graph Neural Network-driven Transformers
• Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling
• A Survey of Visual Transformers
• Creativity and Machine Learning: A Survey
• Dynamic Neural Networks: A Survey
• Transformer-based Models to Deal with Heterogeneous Environments in Human Activity Recognition
• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing
• Are Neural Language Models Good Plagiarists? A Benchmark for Neural Paraphrase Detection
• Characterization of anomalous diffusion through convolutional transformers
• TransformerFusion: Monocular RGB Scene Reconstruction using Transformers
• Towards Visual-Prompt Temporal Answering Grounding in Medical Instructional Video
• Semantic Labeling of High Resolution Images Using EfficientUNets and Transformers
• RPLHR-CT Dataset and Transformer Baseline for Volumetric Super-Resolution from CT Scans
• MPSN: Motion-aware Pseudo Siamese Network for Indoor Video Head Detection in Buildings
• MalBERT: Using Transformers for Cybersecurity and Malicious Software Detection
• Multi-Head Self-Attention via Vision Transformer for Zero-Shot Learning
• Medical Image Segmentation using LeViT-UNet++: A Case Study on GI Tract Data
• Multi-Scale Hybrid Vision Transformer for Learning Gastric Histology: AI-Based Decision Support System for Gastric Cancer Treatment
• Federated Split Vision Transformer for COVID-19 CXR Diagnosis using Task-Agnostic Training
• Exploring Advances in Transformers and CNN for Skin Lesion Diagnosis on Small Datasets
• Unsupervised Person Re-Identification: A Systematic Survey of Challenges and Solutions
• One-Step Abductive Multi-Target Learning with Diverse Noisy Samples and Its Application to Tumour Segmentation for Breast Cancer
• AutoFormer: Searching Transformers for Visual Recognition
• Rethinking Cooking State Recognition with Vision Transformers
• Detection of Illicit Drug Trafficking Events on Instagram: A Deep Multimodal Multilabel Learning Approach
• IntFormer: Predicting pedestrian intention with the aid of the Transformer architecture
• Morphological Classification of Radio Galaxies with wGAN-supported Augmentation
• PU-Transformer: Point Cloud Upsampling Transformer
• PhysFormer: Facial Video-based Physiological Measurement with Temporal Difference Transformer
• Trans-SVNet: Accurate Phase Recognition from Surgical Videos via Hybrid Embedding Aggregation Transformer
• Forensic License Plate Recognition with Compression-Informed Transformers
• On Improving Adversarial Transferability of Vision Transformers
• Deep neural networks approach to microbial colony detection -- a comparative analysis
• End-to-End Trainable Multi-Instance Pose Estimation with Transformers
• Literature review on vulnerability detection using NLP technology
• Vision Transformer using Low-level Chest X-ray Feature Corpus for COVID-19 Diagnosis and Severity Quantification
• Human Image Generation: A Comprehensive Survey
• Performance Evaluation of Action Recognition Models on Low Quality Videos
• DAF:re: A Challenging, Crowd-Sourced, Large-Scale, Long-Tailed Dataset For Anime Character Recognition
• Prototype Memory for Large-scale Face Representation Learning
• Self-supervised Video-centralised Transformer for Video Face Clustering
• Perceptual Image Quality Assessment with Transformers
• Boosting Few-shot Semantic Segmentation with Transformers
• Semi-Supervised Wide-Angle Portraits Correction by Multi-Scale Transformer
• Visual Framing of Science Conspiracy Videos: Integrating Machine Learning with Communication Theories to Study the Use of Color and Brightness
• Investigating Attention Mechanism in 3D Point Cloud Object Detection
• Video Transformer for Deepfake Detection with Incremental Learning
• PnP-3D: A Plug-and-Play for 3D Point Clouds
• A Survey of Fish Tracking Techniques Based on Computer Vision
• Advancing biological super-resolution microscopy through deep learning: a brief review
• Sequential Random Network for Fine-grained Image Classification
• When Liebig's Barrel Meets Facial Landmark Detection: A Practical Model
• Point Cloud Learning with Transformer
• Survey of Visual-Semantic Embedding Methods for Zero-Shot Image Retrieval
• Transformers for prompt-level EMA non-response prediction
• Deep Learning for Anomaly Detection in Railway Systems: A Structured Survey