VOLO: Vision Outlooker for Visual Recognition

Abstract

Visual recognition has been dominated by convolutional neural networks (CNNs) for years. Though recently the prevailing vision transformers (ViTs) have shown great potential of self-attention based models in ImageNet classification, their performance is still inferior to that of the latest SOTA CNNs if no extra data are provided. In this work, we try to close the performance gap and demonstrate that attention-based models are indeed able to outperform CNNs. We find a major factor limiting the performance of ViTs for ImageNet classification is their low efficacy in encoding fine-level features into the token representations. To resolve this, we introduce a novel outlook attention and present a simple and general architecture, termed Vision Outlooker (VOLO). Unlike self-attention that focuses on global dependency modeling at a coarse level, the outlook attention efficiently encodes finer-level features and contexts into tokens, which is shown to be critically beneficial to recognition performance but largely ignored by the self-attention. Experiments show that our VOLO achieves 87.1% top-1 accuracy on ImageNet-1K classification, which is the first model exceeding 87% accuracy on this competitive benchmark, without using any extra training data In addition, the pre-trained VOLO transfers well to downstream tasks, such as semantic segmentation. We achieve 84.3% mIoU score on the cityscapes validation set and 54.3% on the ADE20K validation set. Code is available at \url{https://github.com/sail-sg/volo}.

arXiv commentcode: https://github.com/sail-sg/volo

References in corpus (21)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Distilling the Knowledge in a Neural Network
• Decoupled Weight Decay Regularization
• mixup: Beyond Empirical Risk Minimization
• Language Models are Few-Shot Learners
• Transformer in Transformer
• SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers
• DeepViT: Towards Deeper Vision Transformer
• Searching for Efficient Multi-Scale Architectures for Dense Image Prediction
• High-Performance Large-Scale Image Recognition Without Normalization
• CvT: Introducing Convolutions to Vision Transformers
• All Tokens Matter: Token Labeling for Training Better Vision Transformers
• Are we done with ImageNet?
• Rethinking Spatial Dimensions of Vision Transformers
• Vision Transformers with Patch Diversification
• Refiner: Refining Self-attention for Vision Transformers
• Vision Permutator: A Permutable MLP-Like Architecture for Visual Recognition
• Dilated SpineNet for Semantic Segmentation
• Drawing Multiple Augmentation Samples Per Image During Training Efficiently Decreases Test Error

Cited by in corpus (13)

• A Survey on Visual Transformer
• Attention Mechanisms in Computer Vision: A Survey
• Focal Self-attention for Local-Global Interactions in Vision Transformers
• Masked Autoencoders Are Scalable Vision Learners
• Swin Transformer V2: Scaling Up Capacity and Resolution
• CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation
• A Survey of Visual Transformers
• Efficient Sharpness-aware Minimization for Improved Training of Neural Networks
• An Empirical Study of Training End-to-End Vision-and-Language Transformers
• Exploiting Both Domain-specific and Invariant Knowledge via a Win-win Transformer for Unsupervised Domain Adaptation
• 3rd Place Solution for VisDA 2021 Challenge -- Universally Domain Adaptive Image Recognition
• LUAI Challenge 2021 on Learning to Understand Aerial Images
• Towards Comprehensive Monocular Depth Estimation: Multiple Heads Are Better Than One