Analogous to Evolutionary Algorithm: Designing a Unified Sequence Model
arXiv:2105.15089
Abstract
Inspired by biological evolution, we explain the rationality of Vision Transformer by analogy with the proven practical Evolutionary Algorithm (EA) and derive that both of them have consistent mathematical representation. Analogous to the dynamic local population in EA, we improve the existing transformer structure and propose a more efficient EAT model, and design task-related heads to deal with different tasks more flexibly. Moreover, we introduce the spatial-filling curve into the current vision transformer to sequence image data into a uniform sequential format. Thus we can design a unified EAT framework to address multi-modal tasks, separating the network architecture from the data format adaptation. Our approach achieves state-of-the-art results on the ImageNet classification task compared with recent vision transformer works while having smaller parameters and greater throughput. We further conduct multi-modal tasks to demonstrate the superiority of the unified EAT, e.g., Text-Based Image Retrieval, and our approach improves the rank-1 by +3.7 points over the baseline on the CSS dataset.
References in corpus (21)
- TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation
- Language Models are Few-Shot Learners
- Deformable DETR: Deformable Transformers for End-to-End Object Detection
- Is Space-Time Attention All You Need for Video Understanding?
- Transformer in Transformer
- Evolution Strategies as a Scalable Alternative to Reinforcement Learning
- Linformer: Self-Attention with Linear Complexity
- CoAtNet: Marrying Convolution and Attention for All Data Sizes
- Conditional Positional Encodings for Vision Transformers
- Visual Transformers: Token-based Image Representation and Processing for Computer Vision
- DeepViT: Towards Deeper Vision Transformer
- Reformer: The Efficient Transformer
- LocalViT: Analyzing Locality in Vision Transformers
- TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up
- CvT: Introducing Convolutions to Vision Transformers
- Pre-Trained Image Processing Transformer
- Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth
- Evolution-Guided Policy Gradient in Reinforcement Learning
- LambdaNetworks: Modeling Long-Range Interactions Without Attention
- RealFormer: Transformer Likes Residual Attention
- Space-filling Curves for High-performance Data Mining