Conditional Positional Encodings for Vision Transformers
arXiv:2102.10882
Abstract
We propose a conditional positional encoding (CPE) scheme for vision Transformers. Unlike previous fixed or learnable positional encodings, which are pre-defined and independent of input tokens, CPE is dynamically generated and conditioned on the local neighborhood of the input tokens. As a result, CPE can easily generalize to the input sequences that are longer than what the model has ever seen during training. Besides, CPE can keep the desired translation-invariance in the image classification task, resulting in improved performance. We implement CPE with a simple Position Encoding Generator (PEG) to get seamlessly incorporated into the current Transformer framework. Built on PEG, we present Conditional Position encoding Vision Transformer (CPVT). We demonstrate that CPVT has visually similar attention maps compared to those with learned positional encodings and delivers outperforming results. Our code is available at https://github.com/Meituan-AutoML/CPVT .
Accepted to ICLR2023
References in corpus (3)
Cited by in corpus (41)
- PVT v2: Improved Baselines with Pyramid Vision Transformer
- SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers
- Twins: Revisiting the Design of Spatial Attention in Vision Transformers
- P2T: Pyramid Pooling Transformer for Scene Understanding
- DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification
- Focal Self-attention for Local-Global Interactions in Vision Transformers
- MISSFormer: An Effective Medical Image Segmentation Transformer
- Vision Transformer with Attentive Pooling for Robust Facial Expression Recognition
- ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias
- CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows
- RegionViT: Regional-to-Local Attention for Vision Transformers
- AS-MLP: An Axial Shifted MLP Architecture for Vision
- HIPA: Hierarchical Patch Transformer for Single Image Super Resolution
- CCTrans: Simplifying and Improving Crowd Counting with Transformer
- SOFT: Softmax-free Transformer with Linear Complexity
- CMT: Convolutional Neural Networks Meet Vision Transformers
- A Battle of Network Structures: An Empirical Study of CNN, Transformer, and MLP
- Depth-Wise Convolutions in Vision Transformers for Efficient Training on Small Datasets
- A Survey of Visual Transformers
- Clinically-Inspired Multi-Agent Transformers for Disease Trajectory Forecasting from Multimodal Data
- Dual-stream Network for Visual Recognition
- Glance-and-Gaze Vision Transformer
- 3D Object Tracking with Transformer
- Fully Transformer Networks for Semantic Image Segmentation
- S-MLP: Spatial-Shift MLP Architecture for Vision
- Local-to-Global Self-Attention in Vision Transformers
- Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition
- Building Blocks for a Complex-Valued Transformer Architecture
- OH-Former: Omni-Relational High-Order Transformer for Person Re-Identification
- Analogous to Evolutionary Algorithm: Designing a Unified Sequence Model
- SparX: A Sparse Cross-Layer Connection Mechanism for Hierarchical Vision Mamba and Transformer Networks
- Parameterization of Cross-Token Relations with Relative Positional Encoding for Vision MLP
- CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings
- Study of positional encoding approaches for Audio Spectrogram Transformers
- Hire-MLP: Vision MLP via Hierarchical Rearrangement
- Locally Enhanced Self-Attention: Combining Self-Attention and Convolution as Local and Context Terms
- Convolutional Transformer-Based Image Compression
- Poformer: A simple pooling transformer for speaker verification
- GANSeg: Learning to Segment by Unsupervised Hierarchical Image Generation
- ViTAS: Vision Transformer Architecture Search
- Improving FHB Screening in Wheat Breeding Using an Efficient Transformer Model