Hire-MLP: Vision MLP via Hierarchical Rearrangement
arXiv:2108.13341
Abstract
Previous vision MLPs such as MLP-Mixer and ResMLP accept linearly flattened image patches as input, making them inflexible for different input sizes and hard to capture spatial information. Such approach withholds MLPs from getting comparable performance with their transformer-based counterparts and prevents them from becoming a general backbone for computer vision. This paper presents Hire-MLP, a simple yet competitive vision MLP architecture via \textbf{Hi}erarchical \textbf{re}arrangement, which contains two levels of rearrangements. Specifically, the inner-region rearrangement is proposed to capture local information inside a spatial region, and the cross-region rearrangement is proposed to enable information communication between different regions and capture global context by circularly shifting all tokens along spatial directions. Extensive experiments demonstrate the effectiveness of Hire-MLP as a versatile backbone for various vision tasks. In particular, Hire-MLP achieves competitive results on image classification, object detection and semantic segmentation tasks, e.g., 83.8% top-1 accuracy on ImageNet, 51.7% box AP and 44.8% mask AP on COCO val2017, and 49.9% mIoU on ADE20K, surpassing previous transformer-based and MLP-based models with better trade-off for accuracy and throughput. Code is available at https://github.com/ggjy/Hire-Wave-MLP.pytorch.
References in corpus (23)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Decoupled Weight Decay Regularization
- mixup: Beyond Empirical Risk Minimization
- A Survey on Visual Transformer
- Gaussian Error Linear Units (GELUs)
- PVT v2: Improved Baselines with Pyramid Vision Transformer
- MLP-Mixer: An all-MLP Architecture for Vision
- Transformer in Transformer
- MMDetection: Open MMLab Detection Toolbox and Benchmark
- Conditional Positional Encodings for Vision Transformers
- Early Convolutions Help Transformers See Better
- CvT: Introducing Convolutions to Vision Transformers
- CycleMLP: A MLP-like Architecture for Dense Prediction
- AS-MLP: An Axial Shifted MLP Architecture for Vision
- Pay Attention to MLPs
- S-MLPv2: Improved Spatial-Shift MLP Architecture for Vision
- S-MLP: Spatial-Shift MLP Architecture for Vision
- Vision Permutator: A Permutable MLP-Like Architecture for Visual Recognition
- Global Filter Networks for Image Classification