MLP-Mixer: An all-MLP Architecture for Vision

Abstract

Convolutional Neural Networks (CNNs) are the go-to model for computer vision. Recently, attention-based networks, such as the Vision Transformer, have also become popular. In this paper we show that while convolutions and attention are both sufficient for good performance, neither of them are necessary. We present MLP-Mixer, an architecture based exclusively on multi-layer perceptrons (MLPs). MLP-Mixer contains two types of layers: one with MLPs applied independently to image patches (i.e. "mixing" the per-location features), and one with MLPs applied across patches (i.e. "mixing" spatial information). When trained on large datasets, or with modern regularization schemes, MLP-Mixer attains competitive scores on image classification benchmarks, with pre-training and inference cost comparable to state-of-the-art models. We hope that these results spark further research beyond the realms of well established CNNs and Transformers.

arXiv commentv2: Fixed parameter counts in Table 1. v3: Added results on JFT-3B in Figure 2(right); Added Section 3.4 on the input permutations. v4: Updated the x label in Figure 2(right)

References in corpus (14)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Gaussian Error Linear Units (GELUs)
• Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
• Understanding the Effective Receptive Field in Deep Convolutional Neural Networks
• AutoAugment: Learning Augmentation Policies from Data
• Generating Long Sequences with Sparse Transformers
• Aggregated Residual Transformations for Deep Neural Networks
• Pay Less Attention with Lightweight and Dynamic Convolutions
• High-Performance Large-Scale Image Recognition Without Normalization
• Revisiting ResNets: Improved Training and Scaling Strategies
• Synthesizer: Rethinking Self-Attention in Transformer Models
• A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark
• Are we done with ImageNet?

Cited by in corpus (125)

• A Survey on Visual Transformer
• Attention Mechanisms in Computer Vision: A Survey
• On the Opportunities and Risks of Foundation Models
• ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases
• Efficiently Modeling Long Sequences with Structured State Spaces
• Filter-enhanced MLP is All You Need for Sequential Recommendation
• TSMixer: Lightweight MLP-Mixer Model for Multivariate Time Series Forecasting
• XCiT: Cross-Covariance Image Transformers
• Synthesizer: Rethinking Self-Attention in Transformer Models
• ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias
• CubeMLP: An MLP-based Model for Multimodal Sentiment Analysis and Depression Estimation
• All Tokens Matter: Token Labeling for Training Better Vision Transformers
• IL-MCAM: An interactive learning and multi-channel attention mechanism-based weakly supervised colorectal histopathology image classification approach
• A Practical Survey on Faster and Lighter Transformers
• Do Vision Transformers See Like Convolutional Neural Networks?
• Current Trends in Deep Learning for Earth Observation: An Open-source Benchmark Arena for Image Classification
• CycleMLP: A MLP-like Architecture for Dense Prediction
• AS-MLP: An Axial Shifted MLP Architecture for Vision
• Efficient Self-supervised Vision Transformers for Representation Learning
• ViTGAN: Training GANs with Vision Transformers
• Vision Transformer based COVID-19 Detection using Chest X-rays
• Revisiting the Calibration of Modern Neural Networks
• CrypTen: Secure Multi-Party Computation Meets Machine Learning
• Regular Splitting Graph Network for 3D Human Pose Estimation
• Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
• AutoMLP: Automated MLP for Sequential Recommendations
• Choose a Transformer: Fourier or Galerkin
• A Battle of Network Structures: An Empirical Study of CNN, Transformer, and MLP
• How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers
• RobustART: Benchmarking Robustness on Architecture Design and Training Techniques
• ConvMixer: Feature Interactive Convolution with Curriculum Learning for Small Footprint and Noisy Far-field Keyword Spotting
• Development of Skip Connection in Deep Neural Networks for Computer Vision and Medical Image Analysis: A Survey
• Facial-Sketch Synthesis: A New Challenge
• Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems
• Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillation
• Container: Context Aggregation Network
• Open-Set Recognition: a Good Closed-Set Classifier is All You Need?
• Exploring the Limits of Large Scale Pre-training
• Pay Attention to MLPs
• ResNet strikes back: An improved training procedure in timm
• Deep learning for 3D human pose estimation and mesh recovery: A survey
• S${}^2$-MLPv2: Improved Spatial-Shift MLP Architecture for Vision
• MetaFormer Is Actually What You Need for Vision
• K-Lane: Lidar Lane Dataset and Benchmark for Urban Roads and Highways
• S${}^2$-MLP: Spatial-Shift MLP Architecture for Vision
• UFO: A UniFied TransfOrmer for Vision-Language Representation Learning
• Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers
• Assaying Out-Of-Distribution Generalization in Transfer Learning
• Vision Permutator: A Permutable MLP-Like Architecture for Visual Recognition
• ConvMLP: Hierarchical Convolutional MLPs for Vision
• Supervised Contrastive Learning based Dual-Mixer Model for Remaining Useful Life Prediction
• Photometric Classification of Early-Time Supernova Lightcurves with SCONE
• Exploiting Multiple Sequence Lengths in Fast End to End Training for Image Captioning
• PatchCleanser: Certifiably Robust Defense against Adversarial Patches for Any Image Classifier
• Multi-Scale U-Shape MLP for Hyperspectral Image Classification
• One never walks alone: the effect of the perturber population on subhalo measurements in strong gravitational lenses
• Global Filter Networks for Image Classification
• Image and Model Transformation with Secret Key for Vision Transformer
• Spatial Gated Multi-Layer Perceptron for Land Use and Land Cover Mapping
• Synthesized Speech Detection Using Convolutional Transformer-Based Spectrogram Analysis
• Inductive Biases and Variable Creation in Self-Attention Mechanisms
• PRSeg: A Lightweight Patch Rotate MLP Decoder for Semantic Segmentation
• LegiGPT: Party Politics and Transport Policy with Large Language Model
• Understanding the computational demands underlying visual reasoning
• LordNet: An Efficient Neural Network for Learning to Solve Parametric Partial Differential Equations without Simulated Data
• Axial multi-layer perceptron architecture for automatic segmentation of choroid plexus in multiple sclerosis
• NeRF in detail: Learning to sample for view synthesis
• Hierarchical Associative Memory
• Decision-making and control with diffractive optical networks
• Spatio-temporal Prompting Network for Robust Video Feature Extraction
• TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?
• CrossLinear: Plug-and-Play Cross-Correlation Embedding for Time Series Forecasting with Exogenous Variables
• MLP Singer: Towards Rapid Parallel Korean Singing Voice Synthesis
• On Improving Adversarial Transferability of Vision Transformers
• Beyond First Impressions: Integrating Joint Multi-modal Cues for Comprehensive 3D Representation
• UFO-ViT: High Performance Linear Vision Transformer without Softmax
• MixerGAN: An MLP-Based Architecture for Unpaired Image-to-Image Translation
• Discrete Representations Strengthen Vision Transformer Robustness
• Exploring Corruption Robustness: Inductive Biases in Vision Transformers and MLP-Mixers
• Adversarial Token Attacks on Vision Transformers
• Rethinking Token-Mixing MLP for MLP-based Vision Backbone
• Leveraging Angular Distributions for Improved Knowledge Distillation
• Parameterization of Cross-Token Relations with Relative Positional Encoding for Vision MLP
• Efficient Deep Spiking Multi-Layer Perceptrons with Multiplication-Free Inference
• Characterizing Structural Regularities of Labeled Data in Overparameterized Models
• Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models
• gSwin: Gated MLP Vision Model with Hierarchical Structure of Shifted Window
• Styleformer: Transformer based Generative Adversarial Networks with Style Vector
• Hire-MLP: Vision MLP via Hierarchical Rearrangement
• An Image Patch is a Wave: Phase-Aware Vision MLP
• SCENIC: A JAX Library for Computer Vision Research and Beyond
• PoNet: Pooling Network for Efficient Token Mixing in Long Sequences
• Pyramid Adversarial Training Improves ViT Performance
• Exploring explicit coarse-grained structure in artificial neural networks
• An Enhanced Harmonic Densely Connected Hybrid Transformer Network Architecture for Chronic Wound Segmentation Utilising Multi-Colour Space Tensor Merging
• Coronavirus disease situation analysis and prediction using machine learning: a study on Bangladeshi population
• ImageNet-21K Pretraining for the Masses
• MOI-Mixer: Improving MLP-Mixer with Multi Order Interactions in Sequential Recommendation
• Pointer Value Retrieval: A new benchmark for understanding the limits of neural network generalization
• GenReg: Deep Generative Method for Fast Point Cloud Registration
• Improved Transformer for High-Resolution GANs
• Global and Local Alignment Networks for Unpaired Image-to-Image Translation
• When Medical Imaging Met Self-Attention: A Love Story That Didn't Quite Work Out
• Fast Jet Tagging with MLP-Mixers on FPGAs
• Dynamic Graph Representation Learning for Passenger Behavior Prediction
• Supervised Robustness-preserving Data-free Neural Network Pruning
• HartleyMHA: Self-Attention in Frequency Domain for Resolution-Robust and Parameter-Efficient 3D Image Segmentation
• UniNet: Unified Architecture Search with Convolution, Transformer, and MLP
• Can Attention Enable MLPs To Catch Up With CNNs?
• A remark on a paper of Krotov and Hopfield [arXiv:2008.06996]
• SA-GD: Improved Gradient Descent Learning Strategy with Simulated Annealing
• 3D High-Fidelity Mask Face Presentation Attack Detection Challenge
• Classification of COVID-19 cases from chest CT volumes using hybrid model of 3D CNN and 3D MLP-Mixer
• On the Bias Against Inductive Biases
• Fourier-Mixed Window Attention: Accelerating Informer for Long Sequence Time-Series Forecasting
• PointMixer: MLP-Mixer for Point Cloud Understanding
• Probabilistic Approach for Road-Users Detection
• Contrastive Forward-Forward: A Training Algorithm of Vision Transformer
• Global Interaction Modelling in Vision Transformer via Super Tokens
• KST-Mixer: Kinematic Spatio-Temporal Data Mixer For Colon Shape Estimation
• Understanding Invariance via Feedforward Inversion of Discriminatively Trained Classifiers
• Sifting out the features by pruning: Are convolutional networks the winning lottery ticket of fully connected ones?
• UBoCo : Unsupervised Boundary Contrastive Learning for Generic Event Boundary Detection
• Attention-Free Keyword Spotting
• The Brownian motion in the transformer model