Pay Attention to MLPs

Abstract

Transformers have become one of the most important architectural innovations in deep learning and have enabled many breakthroughs over the past few years. Here we propose a simple network architecture, gMLP, based on MLPs with gating, and show that it can perform as well as Transformers in key language and vision applications. Our comparisons show that self-attention is not critical for Vision Transformers, as gMLP can achieve the same accuracy. For BERT, our model achieves parity with Transformers on pretraining perplexity and is better on some downstream NLP tasks. On finetuning tasks where gMLP performs worse, making the gMLP model substantially larger can close the gap with Transformers. In general, our experiments show that gMLP can scale as well as Transformers over increased data and compute.

References in corpus (13)

• Gaussian Error Linear Units (GELUs)
• MobileNetV2: Inverted Residuals and Linear Bottlenecks
• Scaling Laws for Neural Language Models
• MLP-Mixer: An all-MLP Architecture for Vision
• Pay Less Attention with Lightweight and Dynamic Convolutions
• High-Performance Large-Scale Image Recognition Without Normalization
• Know What You Don't Know: Unanswerable Questions for SQuAD
• Synthesizer: Rethinking Self-Attention in Transformer Models
• CvT: Introducing Convolutions to Vision Transformers
• Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet
• RepMLP: Re-parameterizing Convolutions into Fully-connected Layers for Image Recognition
• LambdaNetworks: Modeling Long-Range Interactions Without Attention
• GLU Variants Improve Transformer

Cited by in corpus (20)

• When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations
• CycleMLP: A MLP-like Architecture for Dense Prediction
• AS-MLP: An Axial Shifted MLP Architecture for Vision
• RepMLP: Re-parameterizing Convolutions into Fully-connected Layers for Image Recognition
• S${}^2$-MLPv2: Improved Spatial-Shift MLP Architecture for Vision
• MetaFormer Is Actually What You Need for Vision
• S${}^2$-MLP: Spatial-Shift MLP Architecture for Vision
• Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers
• ConvMLP: Hierarchical Convolutional MLPs for Vision
• Vision Permutator: A Permutable MLP-Like Architecture for Visual Recognition
• Global Filter Networks for Image Classification
• Visual Transformer with Statistical Test for COVID-19 Classification
• Rethinking Token-Mixing MLP for MLP-based Vision Backbone
• Towards Biologically Plausible Convolutional Networks
• Hire-MLP: Vision MLP via Hierarchical Rearrangement
• An Image Patch is a Wave: Phase-Aware Vision MLP
• MOI-Mixer: Improving MLP-Mixer with Multi Order Interactions in Sequential Recommendation
• RaftMLP: How Much Can Be Done Without Attention and with Less Spatial Locality?
• Convolutional Gated MLP: Combining Convolutions & gMLP
• Attention-Free Keyword Spotting