Vision Permutator: A Permutable MLP-Like Architecture for Visual Recognition

Abstract

In this paper, we present Vision Permutator, a conceptually simple and data efficient MLP-like architecture for visual recognition. By realizing the importance of the positional information carried by 2D feature representations, unlike recent MLP-like models that encode the spatial information along the flattened spatial dimensions, Vision Permutator separately encodes the feature representations along the height and width dimensions with linear projections. This allows Vision Permutator to capture long-range dependencies along one spatial direction and meanwhile preserve precise positional information along the other direction. The resulting position-sensitive outputs are then aggregated in a mutually complementing manner to form expressive representations of the objects of interest. We show that our Vision Permutators are formidable competitors to convolutional neural networks (CNNs) and vision transformers. Without the dependence on spatial convolutions or attention mechanisms, Vision Permutator achieves 81.5% top-1 accuracy on ImageNet without extra large-scale training data (e.g., ImageNet-22k) using only 25M learnable parameters, which is much better than most CNNs and vision transformers under the same model size constraint. When scaling up to 88M, it attains 83.2% top-1 accuracy. We hope this work could encourage research on rethinking the way of encoding spatial information and facilitate the development of MLP-like models. Code is available at https://github.com/Andrew-Qibin/VisionPermutator.

arXiv comment9 pages

References in corpus (14)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• mixup: Beyond Empirical Risk Minimization
• Wide Residual Networks
• MLP-Mixer: An all-MLP Architecture for Vision
• Transformer in Transformer
• DeepViT: Towards Deeper Vision Transformer
• High-Performance Large-Scale Image Recognition Without Normalization
• CvT: Introducing Convolutions to Vision Transformers
• Beyond Self-attention: External Attention using Two Linear Layers for Visual Tasks
• Refiner: Refining Self-attention for Vision Transformers
• Pay Attention to MLPs

Cited by in corpus (12)

• CycleMLP: A MLP-like Architecture for Dense Prediction
• AS-MLP: An Axial Shifted MLP Architecture for Vision
• S${}^2$-MLPv2: Improved Spatial-Shift MLP Architecture for Vision
• MetaFormer Is Actually What You Need for Vision
• ConvMLP: Hierarchical Convolutional MLPs for Vision
• VOLO: Vision Outlooker for Visual Recognition
• PLACE dropout: A Progressive Layer-wise and Channel-wise Dropout for Domain Generalization
• Hire-MLP: Vision MLP via Hierarchical Rearrangement
• An Image Patch is a Wave: Phase-Aware Vision MLP
• RaftMLP: How Much Can Be Done Without Attention and with Less Spatial Locality?
• PointMixer: MLP-Mixer for Point Cloud Understanding
• Global Interaction Modelling in Vision Transformer via Super Tokens