Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions

Abstract

Although using convolutional neural networks (CNNs) as backbones achieves great successes in computer vision, this work investigates a simple backbone network useful for many dense prediction tasks without convolutions. Unlike the recently-proposed Transformer model (e.g., ViT) that is specially designed for image classification, we propose Pyramid Vision Transformer~(PVT), which overcomes the difficulties of porting Transformer to various dense prediction tasks. PVT has several merits compared to prior arts. (1) Different from ViT that typically has low-resolution outputs and high computational and memory cost, PVT can be not only trained on dense partitions of the image to achieve high output resolution, which is important for dense predictions but also using a progressive shrinking pyramid to reduce computations of large feature maps. (2) PVT inherits the advantages from both CNN and Transformer, making it a unified backbone in various vision tasks without convolutions by simply replacing CNN backbones. (3) We validate PVT by conducting extensive experiments, showing that it boosts the performance of many downstream tasks, e.g., object detection, semantic, and instance segmentation. For example, with a comparable number of parameters, RetinaNet+PVT achieves 40.4 AP on the COCO dataset, surpassing RetinNet+ResNet50 (36.3 AP) by 4.1 absolute AP. We hope PVT could serve as an alternative and useful backbone for pixel-level predictions and facilitate future researches. Code is available at https://github.com/whai362/PVT.

arXiv commentAccepted to ICCV 2021

Cited by in corpus (25)

• A Survey on Visual Transformer
• Transformer in Transformer
• A survey of the Vision Transformers and their CNN-Transformer based Variants
• Early Convolutions Help Transformers See Better
• Restormer: Efficient Transformer for High-Resolution Image Restoration
• MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer
• Uformer: A General U-Shaped Transformer for Image Restoration
• CCTrans: Simplifying and Improving Crowd Counting with Transformer
• ViDT: An Efficient and Effective Fully Transformer-based Object Detector
• A Survey of Visual Transformers
• MetaFormer Is Actually What You Need for Vision
• Full-Duplex Strategy for Video Object Segmentation
• Pruning Self-attentions into Convolutional Layers in Single Path
• AdaViT: Adaptive Vision Transformers for Efficient Image Recognition
• FBNetV5: Neural Architecture Search for Multiple Tasks in One Run
• Searching for Efficient Multi-Stage Vision Transformers
• Shunted Self-Attention via Multi-Scale Token Aggregation
• Exploiting Both Domain-specific and Invariant Knowledge via a Win-win Transformer for Unsupervised Domain Adaptation
• A Unified Pruning Framework for Vision Transformers
• Multi-Scale High-Resolution Vision Transformer for Semantic Segmentation
• PatchFormer: An Efficient Point Transformer with Patch Attention
• DS-Net++: Dynamic Weight Slicing for Efficient Inference in CNNs and Transformers
• TransMix: Attend to Mix for Vision Transformers
• Trans4Trans: Efficient Transformer for Transparent Object Segmentation to Help Visually Impaired People Navigate in the Real World
• RaftMLP: How Much Can Be Done Without Attention and with Less Spatial Locality?