CvT: Introducing Convolutions to Vision Transformers

Abstract

We present in this paper a new architecture, named Convolutional vision Transformer (CvT), that improves Vision Transformer (ViT) in performance and efficiency by introducing convolutions into ViT to yield the best of both designs. This is accomplished through two primary modifications: a hierarchy of Transformers containing a new convolutional token embedding, and a convolutional Transformer block leveraging a convolutional projection. These changes introduce desirable properties of convolutional neural networks (CNNs) to the ViT architecture (\ie shift, scale, and distortion invariance) while maintaining the merits of Transformers (\ie dynamic attention, global context, and better generalization). We validate CvT by conducting extensive experiments, showing that this approach achieves state-of-the-art performance over other Vision Transformers and ResNets on ImageNet-1k, with fewer parameters and lower FLOPs. In addition, performance gains are maintained when pretrained on larger datasets (\eg ImageNet-22k) and fine-tuned to downstream tasks. Pre-trained on ImageNet-22k, our CvT-W24 obtains a top-1 accuracy of 87.7\% on the ImageNet-1k val set. Finally, our results show that the positional encoding, a crucial component in existing Vision Transformers, can be safely removed in our model, simplifying the design for higher resolution vision tasks. Code will be released at \url{https://github.com/leoxiaobin/CvT}.

References in corpus (14)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Deep Residual Learning for Image Recognition
• Deformable DETR: Deformable Transformers for End-to-End Object Detection
• Transformer in Transformer
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• Conformer: Convolution-augmented Transformer for Speech Recognition
• Xception: Deep Learning with Depthwise Separable Convolutions
• Pre-Trained Image Processing Transformer
• End-to-End Object Detection with Adaptive Clustering Transformer
• Are we done with ImageNet?
• End-to-End Video Instance Segmentation with Transformers
• Learning Joint Spatial-Temporal Transformations for Video Inpainting
• Evolving Attention with Residual Convolutions

Cited by in corpus (84)

• A Survey on Visual Transformer
• Transformers in Vision: A Survey
• PVT v2: Improved Baselines with Pyramid Vision Transformer
• SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers
• CoAtNet: Marrying Convolution and Attention for All Data Sizes
• Twins: Revisiting the Design of Spatial Attention in Vision Transformers
• A survey of the Vision Transformers and their CNN-Transformer based Variants
• Florence: A New Foundation Model for Computer Vision
• Escaping the Big Data Paradigm with Compact Transformers
• Focal Self-attention for Local-Global Interactions in Vision Transformers
• ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias
• MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer
• All Tokens Matter: Token Labeling for Training Better Vision Transformers
• BiTr-Unet: a CNN-Transformer Combined Network for MRI Brain Tumor Segmentation
• HRFormer: High-Resolution Transformer for Dense Prediction
• Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformer
• Query2Label: A Simple Transformer Way to Multi-Label Classification
• TranSMS: Transformers for Super-Resolution Calibration in Magnetic Particle Imaging
• Uformer: A General U-Shaped Transformer for Image Restoration
• CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows
• CycleMLP: A MLP-like Architecture for Dense Prediction
• RegionViT: Regional-to-Local Attention for Vision Transformers
• LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference
• Efficient Self-supervised Vision Transformers for Representation Learning
• Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet
• IA-RED${}^2$: Interpretability-Aware Redundancy Reduction for Vision Transformers
• CMT: Convolutional Neural Networks Meet Vision Transformers
• A Battle of Network Structures: An Empirical Study of CNN, Transformer, and MLP
• Long-Short Transformer: Efficient Transformers for Language and Vision
• How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers
• Vision Transformers with Patch Diversification
• Dual-stream Network for Visual Recognition
• Pay Attention to MLPs
• Glance-and-Gaze Vision Transformer
• Fully Transformer Networks for Semantic Image Segmentation
• MetaFormer Is Actually What You Need for Vision
• S${}^2$-MLP: Spatial-Shift MLP Architecture for Vision
• Vision Permutator: A Permutable MLP-Like Architecture for Visual Recognition
• VOLO: Vision Outlooker for Visual Recognition
• X-volution: On the unification of convolution and self-attention
• Local-to-Global Self-Attention in Vision Transformers
• MVT: Multi-view Vision Transformer for 3D Object Recognition
• Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition
• Global Filter Networks for Image Classification
• Visual Parser: Representing Part-whole Hierarchies with Transformers
• AutoFormer: Searching Transformers for Visual Recognition
• AdaViT: Adaptive Vision Transformers for Efficient Image Recognition
• Homogeneous vector bundles and $G$-equivariant convolutional neural networks
• Analogous to Evolutionary Algorithm: Designing a Unified Sequence Model
• Searching for Efficient Multi-Stage Vision Transformers
• Vision Xformers: Efficient Attention for Image Classification
• UFO-ViT: High Performance Linear Vision Transformer without Softmax
• Transformers predicting the future. Applying attention in next-frame and time series forecasting
• Primer: Searching for Efficient Transformers for Language Modeling
• MSG-Transformer: Exchanging Local Spatial Information by Manipulating Messenger Tokens
• Towards Robust Vision Transformer
• On the Integration of Self-Attention and Convolution
• Rethinking Token-Mixing MLP for MLP-based Vision Backbone
• Adversarial Token Attacks on Vision Transformers
• Shifted Chunk Transformer for Spatio-Temporal Representational Learning
• CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings
• Styleformer: Transformer based Generative Adversarial Networks with Style Vector
• More than Encoder: Introducing Transformer Decoder to Upsample
• Hire-MLP: Vision MLP via Hierarchical Rearrangement
• KVT: k-NN Attention for Boosting Vision Transformers
• Locally Enhanced Self-Attention: Combining Self-Attention and Convolution as Local and Context Terms
• VidTr: Video Transformer Without Convolutions
• 3rd Place Solution to Google Landmark Recognition Competition 2021
• Transformed CNNs: recasting pre-trained convolutional layers with self-attention
• NomMer: Nominate Synergistic Context in Vision Transformer for Visual Recognition
• CAE-Transformer: Transformer-based Model to Predict Invasiveness of Lung Adenocarcinoma Subsolid Nodules from Non-thin Section 3D CT Scans
• UniNet: Unified Architecture Search with Convolution, Transformer, and MLP
• Concurrent Neural Tree and Data Preprocessing AutoML for Image Classification
• TED-net: Convolution-free T2T Vision Transformer-based Encoder-decoder Dilation network for Low-dose CT Denoising
• Armour: Generalizable Compact Self-Attention for Vision Transformers
• Convolutional Gated MLP: Combining Convolutions & gMLP
• ViTAS: Vision Transformer Architecture Search
• Semi-Supervised Vision Transformers
• Searching for TrioNet: Combining Convolution with Local and Global Self-Attention
• CpT: Convolutional Point Transformer for 3D Point Cloud Processing
• Vision Pair Learning: An Efficient Training Framework for Image Classification
• HindSight: A Graph-Based Vision Model Architecture For Representing Part-Whole Hierarchies
• Audiomer: A Convolutional Transformer For Keyword Spotting
• Ripple Attention for Visual Perception with Sub-quadratic Complexity