Vision Transformers with Patch Diversification

Abstract

Vision transformer has demonstrated promising performance on challenging computer vision tasks. However, directly training the vision transformers may yield unstable and sub-optimal results. Recent works propose to improve the performance of the vision transformers by modifying the transformer structures, e.g., incorporating convolution layers. In contrast, we investigate an orthogonal approach to stabilize the vision transformer training without modifying the networks. We observe the instability of the training can be attributed to the significant similarity across the extracted patch representations. More specifically, for deep vision transformers, the self-attention blocks tend to map different patches into similar latent representations, yielding information loss and performance degradation. To alleviate this problem, in this work, we introduce novel loss functions in vision transformer training to explicitly encourage diversity across patch representations for more discriminative feature extraction. We empirically show that our proposed techniques stabilize the training and allow us to train wider and deeper vision transformers. We further show the diversified features significantly benefit the downstream tasks in transfer learning. For semantic segmentation, we enhance the state-of-the-art (SOTA) results on Cityscapes and ADE20k. Our code is available at https://github.com/ChengyueGongR/PatchVisionTransformer.

arXiv commentpreprint

References in corpus (17)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Decoupled Weight Decay Regularization
• mixup: Beyond Empirical Risk Minimization
• Deformable DETR: Deformable Transformers for End-to-End Object Detection
• Is Space-Time Attention All You Need for Video Understanding?
• EfficientNetV2: Smaller Models and Faster Training
• Transformer in Transformer
• DeepViT: Towards Deeper Vision Transformer
• TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up
• High-Performance Large-Scale Image Recognition Without Normalization
• CvT: Introducing Convolutions to Vision Transformers
• GraphNorm: A Principled Approach to Accelerating Graph Neural Network Training
• Multiscale Deep Equilibrium Models
• Talking-Heads Attention
• MaxUp: A Simple Way to Improve Generalization of Neural Network Training
• Improving Graph Attention Networks with Large Margin-based Constraints
• Learning towards Minimum Hyperspherical Energy

Cited by in corpus (9)

• Focal Self-attention for Local-Global Interactions in Vision Transformers
• Long-Short Transformer: Efficient Transformers for Language and Vision
• A Survey of Visual Transformers
• Refiner: Refining Self-attention for Vision Transformers
• Gastrointestinal Disorder Detection with a Transformer Based Approach
• VOLO: Vision Outlooker for Visual Recognition
• Homogeneous Tokenizer Matters: Homogeneous Visual Tokenizer for Remote Sensing Image Understanding
• Searching for Efficient Multi-Stage Vision Transformers
• KVT: k-NN Attention for Boosting Vision Transformers