MLP-Mixer: An all-MLP Architecture for Vision

Abstract

Convolutional Neural Networks (CNNs) are the go-to model for computer vision. Recently, attention-based networks, such as the Vision Transformer, have also become popular. In this paper we show that while convolutions and attention are both sufficient for good performance, neither of them are necessary. We present MLP-Mixer, an architecture based exclusively on multi-layer perceptrons (MLPs). MLP-Mixer contains two types of layers: one with MLPs applied independently to image patches (i.e. "mixing" the per-location features), and one with MLPs applied across patches (i.e. "mixing" spatial information). When trained on large datasets, or with modern regularization schemes, MLP-Mixer attains competitive scores on image classification benchmarks, with pre-training and inference cost comparable to state-of-the-art models. We hope that these results spark further research beyond the realms of well established CNNs and Transformers.

v2: Fixed parameter counts in Table 1. v3: Added results on JFT-3B in Figure 2(right); Added Section 3.4 on the input permutations. v4: Updated the x label in Figure 2(right)

References in corpus (8)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Understanding the Effective Receptive Field in Deep Convolutional Neural Networks
• Generating Long Sequences with Sparse Transformers
• Pay Less Attention with Lightweight and Dynamic Convolutions
• High-Performance Large-Scale Image Recognition Without Normalization
• Revisiting ResNets: Improved Training and Scaling Strategies
• A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark
• Are we done with ImageNet?

Cited by in corpus (33)

• Filter-enhanced MLP is All You Need for Sequential Recommendation
• XCiT: Cross-Covariance Image Transformers
• CubeMLP: An MLP-based Model for Multimodal Sentiment Analysis and Depression Estimation
• Revisiting the Calibration of Modern Neural Networks
• Container: Context Aggregation Network
• Exploring the Limits of Large Scale Pre-training
• Pay Attention to MLPs
• ResNet strikes back: An improved training procedure in timm
• S${}^2$-MLPv2: Improved Spatial-Shift MLP Architecture for Vision
• S${}^2$-MLP: Spatial-Shift MLP Architecture for Vision
• Assaying Out-Of-Distribution Generalization in Transfer Learning
• Vision Permutator: A Permutable MLP-Like Architecture for Visual Recognition
• ConvMLP: Hierarchical Convolutional MLPs for Vision
• Global Filter Networks for Image Classification
• Synthesized Speech Detection Using Convolutional Transformer-Based Spectrogram Analysis
• NeRF in detail: Learning to sample for view synthesis
• Hierarchical Associative Memory
• UFO-ViT: High Performance Linear Vision Transformer without Softmax
• Adversarial Token Attacks on Vision Transformers
• Rethinking Token-Mixing MLP for MLP-based Vision Backbone
• Exploring Corruption Robustness: Inductive Biases in Vision Transformers and MLP-Mixers
• Parameterization of Cross-Token Relations with Relative Positional Encoding for Vision MLP
• Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models
• SCENIC: A JAX Library for Computer Vision Research and Beyond
• MOI-Mixer: Improving MLP-Mixer with Multi Order Interactions in Sequential Recommendation
• Coronavirus disease situation analysis and prediction using machine learning: a study on Bangladeshi population
• Can Attention Enable MLPs To Catch Up With CNNs?
• SA-GD: Improved Gradient Descent Learning Strategy with Simulated Annealing
• A remark on a paper of Krotov and Hopfield [arXiv:2008.06996]
• UniNet: Unified Architecture Search with Convolution, Transformer, and MLP
• On the Bias Against Inductive Biases
• 3D High-Fidelity Mask Face Presentation Attack Detection Challenge
• The Brownian motion in the transformer model