UniNeXt: Exploring A Unified Architecture for Vision Recognition

Abstract

Vision Transformers have shown great potential in computer vision tasks. Most recent works have focused on elaborating the spatial token mixer for performance gains. However, we observe that a well-designed general architecture can significantly improve the performance of the entire backbone, regardless of which spatial token mixer is equipped. In this paper, we propose UniNeXt, an improved general architecture for the vision backbone. To verify its effectiveness, we instantiate the spatial token mixer with various typical and modern designs, including both convolution and attention modules. Compared with the architecture in which they are first proposed, our UniNeXt architecture can steadily boost the performance of all the spatial token mixers, and narrows the performance gap among them. Surprisingly, our UniNeXt equipped with naive local window attention even outperforms the previous state-of-the-art. Interestingly, the ranking of these spatial token mixers also changes under our UniNeXt, suggesting that an excellent spatial token mixer may be stifled due to a suboptimal general architecture, which further shows the importance of the study on the general architecture of vision backbone.

arXiv commentAccep by ACM MM 2023

References in corpus (16)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Decoupled Weight Decay Regularization
• PVT v2: Improved Baselines with Pyramid Vision Transformer
• SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers
• Twins: Revisiting the Design of Spatial Attention in Vision Transformers
• Focal Self-attention for Local-Global Interactions in Vision Transformers
• Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
• Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformer
• CycleMLP: A MLP-like Architecture for Dense Prediction
• Mobile-Former: Bridging MobileNet and Transformer
• Fully Transformer Networks for Semantic Image Segmentation
• PRSeg: A Lightweight Patch Rotate MLP Decoder for Semantic Segmentation
• Dynamic Group Transformer: A General Vision Transformer Backbone with Dynamic Group Attention
• Exploring vision transformer layer choosing for semantic segmentation
• AxWin Transformer: A Context-Aware Vision Transformer Backbone with Axial Windows