Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet

Abstract

The strong performance of vision transformers on image classification and other vision tasks is often attributed to the design of their multi-head attention layers. However, the extent to which attention is responsible for this strong performance remains unclear. In this short report, we ask: is the attention layer even necessary? Specifically, we replace the attention layer in a vision transformer with a feed-forward layer applied over the patch dimension. The resulting architecture is simply a series of feed-forward layers applied over the patch and feature dimensions in an alternating fashion. In experiments on ImageNet, this architecture performs surprisingly well: a ViT/DeiT-base-sized model obtains 74.9\% top-1 accuracy, compared to 77.9\% and 79.9\% for ViT and DeiT respectively. These results indicate that aspects of vision transformers other than attention, such as the patch embedding, may be more responsible for their strong performance than previously thought. We hope these results prompt the community to spend more time trying to understand why our current models are as effective as they are.

arXiv commentShort Technical Report. GitHub: https://github.com/lukemelas/do-you-even-need-attention

References in corpus (3)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• CvT: Introducing Convolutions to Vision Transformers
• LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference

Cited by in corpus (20)

• A Survey on Visual Transformer
• Transformers in Vision: A Survey
• XCiT: Cross-Covariance Image Transformers
• ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias
• When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations
• AS-MLP: An Axial Shifted MLP Architecture for Vision
• A Battle of Network Structures: An Empirical Study of CNN, Transformer, and MLP
• Graph-MLP: Node Classification without Message Passing in Graph
• Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillation
• Open-Set Recognition: a Good Closed-Set Classifier is All You Need?
• Pay Attention to MLPs
• S${}^2$-MLP: Spatial-Shift MLP Architecture for Vision
• Axial multi-layer perceptron architecture for automatic segmentation of choroid plexus in multiple sclerosis
• Hierarchical Associative Memory
• Rethinking Token-Mixing MLP for MLP-based Vision Backbone
• Characterizing Structural Regularities of Labeled Data in Overparameterized Models
• Can Attention Enable MLPs To Catch Up With CNNs?
• A remark on a paper of Krotov and Hopfield [arXiv:2008.06996]
• RaftMLP: How Much Can Be Done Without Attention and with Less Spatial Locality?
• Attention-Free Keyword Spotting