UFO-ViT: High Performance Linear Vision Transformer without Softmax

Abstract

Vision transformers have become one of the most important models for computer vision tasks. Although they outperform prior works, they require heavy computational resources on a scale that is quadratic to $N$. This is a major drawback of the traditional self-attention (SA) algorithm. Here, we propose the Unit Force Operated Vision Transformer (UFO-ViT), a novel SA mechanism that has linear complexity. The main approach of this work is to eliminate nonlinearity from the original SA. We factorize the matrix multiplication of the SA mechanism without complicated linear approximation. By modifying only a few lines of code from the original SA, the proposed models outperform most transformer-based models on image classification and dense prediction tasks on most capacity regimes.

References in corpus (14)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• MLP-Mixer: An all-MLP Architecture for Vision
• Linformer: Self-Attention with Linear Complexity
• MMDetection: Open MMLab Detection Toolbox and Benchmark
• Large Batch Training of Convolutional Networks
• Generating Long Sequences with Sparse Transformers
• Reformer: The Efficient Transformer
• Escaping the Big Data Paradigm with Compact Transformers
• XCiT: Cross-Covariance Image Transformers
• CvT: Introducing Convolutions to Vision Transformers
• Rethinking Attention with Performers
• LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference
• Rethinking Spatial Dimensions of Vision Transformers
• Nyströmformer: A Nyström-Based Algorithm for Approximating Self-Attention