Can Vision Transformers Perform Convolution?

Abstract

Several recent studies have demonstrated that attention-based networks, such as Vision Transformer (ViT), can outperform Convolutional Neural Networks (CNNs) on several computer vision tasks without using convolutional layers. This naturally leads to the following questions: Can a self-attention layer of ViT express any convolution operation? In this work, we prove that a single ViT layer with image patches as the input can perform any convolution operation constructively, where the multi-head attention mechanism and the relative positional encoding play essential roles. We further provide a lower bound on the number of heads for Vision Transformers to express CNNs. Corresponding with our analysis, experimental results show that the construction in our proof can help inject convolutional bias into Transformers and significantly improve the performance of ViT in low data regimes.

References in corpus (9)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Big Bird: Transformers for Longer Sequences
• On Layer Normalization in the Transformer Architecture
• When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations
• Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth
• Are Transformers universal approximators of sequence-to-sequence functions?
• The Depth-to-Width Interplay in Self-Attention
• Which transformer architecture fits my data? A vocabulary bottleneck in self-attention
• On the Expressive Power of Self-Attention Matrices