Parameterization of Cross-Token Relations with Relative Positional Encoding for Vision MLP

Abstract

Vision multi-layer perceptrons (MLPs) have shown promising performance in computer vision tasks, and become the main competitor of CNNs and vision Transformers. They use token-mixing layers to capture cross-token interactions, as opposed to the multi-head self-attention mechanism used by Transformers. However, the heavily parameterized token-mixing layers naturally lack mechanisms to capture local information and multi-granular non-local relations, thus their discriminative power is restrained. To tackle this issue, we propose a new positional spacial gating unit (PoSGU). It exploits the attention formulations used in the classical relative positional encoding (RPE), to efficiently encode the cross-token relations for token mixing. It can successfully reduce the current quadratic parameter complexity $O(N^2)$ of vision MLPs to $O(N)$ and $O(1)$. We experiment with two RPE mechanisms, and further propose a group-wise extension to improve their expressive power with the accomplishment of multi-granular contexts. These then serve as the key building blocks of a new type of vision MLP, referred to as PosMLP. We evaluate the effectiveness of the proposed approach by conducting thorough experiments, demonstrating an improved or comparable performance with reduced parameter complexity. For instance, for a model trained on ImageNet1K, we achieve a performance improvement from 72.14\% to 74.02\% and a learnable parameter reduction from $19.4M$ to $18.2M$. Code could be found at https://github.com/Zhicaiwww/PosMLP.

References in corpus (12)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Distilling the Knowledge in a Neural Network
• Decoupled Weight Decay Regularization
• mixup: Beyond Empirical Risk Minimization
• MLP-Mixer: An all-MLP Architecture for Vision
• ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases
• CoAtNet: Marrying Convolution and Attention for All Data Sizes
• Conditional Positional Encodings for Vision Transformers
• Early Convolutions Help Transformers See Better
• CycleMLP: A MLP-like Architecture for Dense Prediction
• RepMLP: Re-parameterizing Convolutions into Fully-connected Layers for Image Recognition
• CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings

Cited by in corpus (1)

• PosMLP-Video: Spatial and Temporal Relative Position Encoding for Efficient Video Recognition