Augmented Shortcuts for Vision Transformers

Abstract

Transformer models have achieved great progress on computer vision tasks recently. The rapid development of vision transformers is mainly contributed by their high representation ability for extracting informative features from input images. However, the mainstream transformer models are designed with deep architectures, and the feature diversity will be continuously reduced as the depth increases, i.e., feature collapse. In this paper, we theoretically analyze the feature collapse phenomenon and study the relationship between shortcuts and feature diversity in these transformer models. Then, we present an augmented shortcut scheme, which inserts additional paths with learnable parameters in parallel on the original shortcuts. To save the computational costs, we further explore an efficient approach that uses the block-circulant projection to implement augmented shortcuts. Extensive experiments conducted on benchmark datasets demonstrate the effectiveness of the proposed method, which brings about 1% accuracy increase of the state-of-the-art visual transformers without obviously increasing their parameters and FLOPs.

References in corpus (10)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• LSTM: A Search Space Odyssey
• mixup: Beyond Empirical Risk Minimization
• Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
• FractalNet: Ultra-Deep Neural Networks without Residuals
• Residual Networks Behave Like Ensembles of Relatively Shallow Networks
• Pre-Trained Image Processing Transformer
• The Shattered Gradients Problem: If resnets are the answer, then what is the question?
• Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth
• Towards Understanding the Importance of Shortcut Connections in Residual Networks