ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias

Abstract

Transformers have shown great potential in various computer vision tasks owing to their strong capability in modeling long-range dependency using the self-attention mechanism. Nevertheless, vision transformers treat an image as 1D sequence of visual tokens, lacking an intrinsic inductive bias (IB) in modeling local visual structures and dealing with scale variance. Alternatively, they require large-scale training data and longer training schedules to learn the IB implicitly. In this paper, we propose a novel Vision Transformer Advanced by Exploring intrinsic IB from convolutions, ie, ViTAE. Technically, ViTAE has several spatial pyramid reduction modules to downsample and embed the input image into tokens with rich multi-scale context by using multiple convolutions with different dilation rates. In this way, it acquires an intrinsic scale invariance IB and is able to learn robust feature representation for objects at various scales. Moreover, in each transformer layer, ViTAE has a convolution block in parallel to the multi-head self-attention module, whose features are fused and fed into the feed-forward network. Consequently, it has the intrinsic locality IB and is able to learn local features and global dependencies collaboratively. Experiments on ImageNet as well as downstream tasks prove the superiority of ViTAE over the baseline transformer and concurrent works. Source code and pretrained models will be available at GitHub.

Accepted by NeurIPS 2021, 23 pages, including downstream task results (detection, semantic segmentation, human pose, and video object segmentation)

References in corpus (15)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Distilling the Knowledge in a Neural Network
• Rethinking Atrous Convolution for Semantic Image Segmentation
• MLP-Mixer: An all-MLP Architecture for Vision
• Transformer in Transformer
• Understanding the Effective Receptive Field in Deep Convolutional Neural Networks
• Conditional Positional Encodings for Vision Transformers
• LocalViT: Analyzing Locality in Vision Transformers
• CvT: Introducing Convolutions to Vision Transformers
• LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference
• Beyond Self-attention: External Attention using Two Linear Layers for Visual Tasks
• Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet
• Rethinking Spatial Dimensions of Vision Transformers
• RepMLP: Re-parameterizing Convolutions into Fully-connected Layers for Image Recognition

Cited by in corpus (16)

• HANet: A Hierarchical Attention Network for Change Detection With Bitemporal Very-High-Resolution Remote Sensing Images
• An Empirical Study of Remote Sensing Pretraining
• Change Guiding Network: Incorporating Change Prior to Guide Change Detection in Remote Sensing Imagery
• TransCAM: Transformer Attention-based CAM Refinement for Weakly Supervised Semantic Segmentation
• DCN-T: Dual Context Network with Transformer for Hyperspectral Image Classification
• DSP: Dual Soft-Paste for Unsupervised Domain Adaptive Semantic Segmentation
• Exploring the Synergies of Hybrid CNNs and ViTs Architectures for Computer Vision: A survey
• Depth-Wise Convolutions in Vision Transformers for Efficient Training on Small Datasets
• HRPVT: High-Resolution Pyramid Vision Transformer for medium and small-scale human pose estimation
• Transformer-based Context Condensation for Boosting Feature Pyramids in Object Detection
• PVG: Progressive Vision Graph for Vision Recognition
• Pruning Self-attentions into Convolutional Layers in Single Path
• Information-Theoretic Odometry Learning
• Convolutional Gated MLP: Combining Convolutions & gMLP
• The USYD-JD Speech Translation System for IWSLT 2021
• Ripple Attention for Visual Perception with Sub-quadratic Complexity