DeepViT: Towards Deeper Vision Transformer

Abstract

Vision transformers (ViTs) have been successfully applied in image classification tasks recently. In this paper, we show that, unlike convolution neural networks (CNNs)that can be improved by stacking more convolutional layers, the performance of ViTs saturate fast when scaled to be deeper. More specifically, we empirically observe that such scaling difficulty is caused by the attention collapse issue: as the transformer goes deeper, the attention maps gradually become similar and even much the same after certain layers. In other words, the feature maps tend to be identical in the top layers of deep ViT models. This fact demonstrates that in deeper layers of ViTs, the self-attention mechanism fails to learn effective concepts for representation learning and hinders the model from getting expected performance gain. Based on above observation, we propose a simple yet effective method, named Re-attention, to re-generate the attention maps to increase their diversity at different layers with negligible computation and memory cost. The pro-posed method makes it feasible to train deeper ViT models with consistent performance improvements via minor modification to existing ViT models. Notably, when training a deep ViT model with 32 transformer blocks, the Top-1 classification accuracy can be improved by 1.6% on ImageNet. Code is publicly available at https://github.com/zhoudaquan/dvit_repo.

References in corpus (13)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• mixup: Beyond Empirical Risk Minimization
• Language Models are Few-Shot Learners
• Wide Residual Networks
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Axial Attention in Multidimensional Transformers
• Coordinate Attention for Efficient Mobile Network Design
• MixConv: Mixed Depthwise Convolutional Kernels
• Stand-Alone Self-Attention in Vision Models
• ConvBERT: Improving BERT with Span-based Dynamic Convolution
• Pre-Trained Image Processing Transformer

Cited by in corpus (36)

• A Survey on Visual Transformer
• Transformers in Vision: A Survey
• Attention Mechanisms in Computer Vision: A Survey
• CoAtNet: Marrying Convolution and Attention for All Data Sizes
• DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification
• Focal Self-attention for Local-Global Interactions in Vision Transformers
• Vision Transformer with Attentive Pooling for Robust Facial Expression Recognition
• MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer
• All Tokens Matter: Token Labeling for Training Better Vision Transformers
• SpeechFormer++: A Hierarchical Efficient Framework for Paralinguistic Speech Processing
• AS-MLP: An Axial Shifted MLP Architecture for Vision
• Chasing Sparsity in Vision Transformers: An End-to-End Exploration
• Vision Transformers with Patch Diversification
• Interpretable CNN-Multilevel Attention Transformer for Rapid Recognition of Pneumonia from Chest X-Ray Images
• A Survey of Visual Transformers
• Refiner: Refining Self-attention for Vision Transformers
• Dual-stream Network for Visual Recognition
• Vision Permutator: A Permutable MLP-Like Architecture for Visual Recognition
• VOLO: Vision Outlooker for Visual Recognition
• TempEE: Temporal-Spatial Parallel Transformer for Radar Echo Extrapolation Beyond Auto-Regression
• Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition
• Knowledge Amalgamation for Object Detection with Transformers
• Pruning Self-attentions into Convolutional Layers in Single Path
• Efficient Scopeformer: Towards Scalable and Rich Feature Extraction for Intracranial Hemorrhage Detection
• Analogous to Evolutionary Algorithm: Designing a Unified Sequence Model
• Weakly Supervised Intracranial Hemorrhage Segmentation using Head-Wise Gradient-Infused Self-Attention Maps from a Swin Transformer in Categorical Learning
• Shunted Self-Attention via Multi-Scale Token Aggregation
• SDNet: mutil-branch for single image deraining using swin
• Towards Training Stronger Video Vision Transformers for EPIC-KITCHENS-100 Action Recognition
• Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks
• Self-supervised Video-centralised Transformer for Video Face Clustering
• More than Encoder: Introducing Transformer Decoder to Upsample
• KVT: k-NN Attention for Boosting Vision Transformers
• A Stronger Baseline for Ego-Centric Action Detection
• Architecture Analysis and Benchmarking of 3D U-shaped Deep Learning Models for Thoracic Anatomical Segmentation
• NeuraCrypt: Hiding Private Health Data via Random Neural Networks for Public Training