SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers

Abstract

We present SegFormer, a simple, efficient yet powerful semantic segmentation framework which unifies Transformers with lightweight multilayer perception (MLP) decoders. SegFormer has two appealing features: 1) SegFormer comprises a novel hierarchically structured Transformer encoder which outputs multiscale features. It does not need positional encoding, thereby avoiding the interpolation of positional codes which leads to decreased performance when the testing resolution differs from training. 2) SegFormer avoids complex decoders. The proposed MLP decoder aggregates information from different layers, and thus combining both local attention and global attention to render powerful representations. We show that this simple and lightweight design is the key to efficient segmentation on Transformers. We scale our approach up to obtain a series of models from SegFormer-B0 to SegFormer-B5, reaching significantly better performance and efficiency than previous counterparts. For example, SegFormer-B4 achieves 50.3% mIoU on ADE20K with 64M parameters, being 5x smaller and 2.2% better than the previous best method. Our best model, SegFormer-B5, achieves 84.0% mIoU on Cityscapes validation set and shows excellent zero-shot robustness on Cityscapes-C. Code will be released at: github.com/NVlabs/SegFormer.

arXiv commentAccepted by NeurIPS 2021

References in corpus (20)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Learning Transferable Visual Models From Natural Language Supervision
• Understanding the Effective Receptive Field in Deep Convolutional Neural Networks
• DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs
• OCNet: Object Context Network for Scene Parsing
• Conditional Positional Encodings for Vision Transformers
• TransTrack: Multiple Object Tracking with Transformer
• Pyramid Attention Network for Semantic Segmentation
• ContextNet: Exploring Context and Detail for Semantic Segmentation in Real-time
• CvT: Introducing Convolutions to Vision Transformers
• Training Vision Transformers for Image Retrieval
• FasterSeg: Searching for Faster Real-time Semantic Segmentation
• TransReID: Transformer-based Object Re-Identification
• LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inference
• Learning a Discriminative Feature Network for Semantic Segmentation
• CGNet: A Light-weight Context Guided Network for Semantic Segmentation
• Segmenting Transparent Object in the Wild with Transformer
• Joint Semantic Segmentation and Boundary Detection using Iterative Pyramid Contexts
• Colorization Transformer

Cited by in corpus (38)

• A Survey on Visual Transformer
• Segment Anything Model for Medical Image Analysis: an Experimental Study
• Focal Self-attention for Local-Global Interactions in Vision Transformers
• MISSFormer: An Effective Medical Image Segmentation Transformer
• Localizing Objects with Self-Supervised Transformers and no Labels
• CycleMLP: A MLP-like Architecture for Dense Prediction
• Part-guided Relational Transformers for Fine-grained Visual Recognition
• Tomato Maturity Recognition with Convolutional Transformers
• 6D-ViT: Category-Level 6D Object Pose Estimation via Transformer-based Instance Representation Learning
• A Survey of Visual Transformers
• Globally Scalable Glacier Mapping by Deep Learning Matches Expert Delineation Accuracy
• End-to-end deep learning for directly estimating grape yield from ground-based imagery
• Assessment of a new GeoAI foundation model for flood inundation mapping
• SwinCross: Cross-modal Swin Transformer for Head-and-Neck Tumor Segmentation in PET/CT Images
• Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers
• Deep Learning-based Bathymetry Retrieval without In-situ Depths using Remote Sensing Imagery and SfM-MVS DSMs with Data Gaps
• A large-scale climate-aware satellite image dataset for domain adaptive land-cover semantic segmentation
• VOLO: Vision Outlooker for Visual Recognition
• WildScenes: A Benchmark for 2D and 3D Semantic Segmentation in Large-scale Natural Environments
• Single-Temporal Supervised Learning for Universal Remote Sensing Change Detection
• OH-Former: Omni-Relational High-Order Transformer for Person Re-Identification
• Timeline and Boundary Guided Diffusion Network for Video Shadow Detection
• UniNeXt: Exploring A Unified Architecture for Vision Recognition
• SegDiff: Image Segmentation with Diffusion Probabilistic Models
• FBNetV5: Neural Architecture Search for Multiple Tasks in One Run
• Co-Training for Unsupervised Domain Adaptation of Semantic Segmentation Models
• Pseudolabel guided pixels contrast for domain adaptive semantic segmentation
• HSVLT: Hierarchical Scale-Aware Vision-Language Transformer for Multi-Label Image Classification
• Enhancing kelp forest detection in remote sensing images using crowdsourced labels with Mixed Vision Transformers and ConvNeXt segmentation models
• PEAR: Phrase-Based Hand-Object Interaction Anticipation
• Channelized Axial Attention for Semantic Segmentation -- Considering Channel Relation within Spatial Attention for Semantic Segmentation
• Transformer for Polyp Detection
• Calibration & Reconstruction: Deep Integrated Language for Referring Image Segmentation
• AMBER -- Advanced SegFormer for Multi-Band Image Segmentation: an application to Hyperspectral Imaging
• Semantic Segmentation on VSPW Dataset through Aggregation of Transformer Models
• Unsupervised Industrial Anomaly Detection via Pattern Generative and Contrastive Networks
• LUAI Challenge 2021 on Learning to Understand Aerial Images
• Quotient Network -- A Network Similar to ResNet but Learning Quotients