Per-Pixel Classification is Not All You Need for Semantic Segmentation

Abstract

Modern approaches typically formulate semantic segmentation as a per-pixel classification task, while instance-level segmentation is handled with an alternative mask classification. Our key insight: mask classification is sufficiently general to solve both semantic- and instance-level segmentation tasks in a unified manner using the exact same model, loss, and training procedure. Following this observation, we propose MaskFormer, a simple mask classification model which predicts a set of binary masks, each associated with a single global class label prediction. Overall, the proposed mask classification-based method simplifies the landscape of effective approaches to semantic and panoptic segmentation tasks and shows excellent empirical results. In particular, we observe that MaskFormer outperforms per-pixel classification baselines when the number of classes is large. Our mask classification-based method outperforms both current state-of-the-art semantic (55.6 mIoU on ADE20K) and panoptic segmentation (52.7 PQ on COCO) models.

NeurIPS 2021, Spotlight. Project page: https://bowenc0221.github.io/maskformer

References in corpus (5)

• Decoupled Weight Decay Regularization
• Rethinking Atrous Convolution for Semantic Image Segmentation
• Convolutional Feature Masking for Joint Object and Stuff Segmentation
• Hierarchical Multi-Scale Attention for Semantic Segmentation
• SPGNet: Semantic Prediction Guidance for Scene Parsing

Cited by in corpus (12)

• A Survey of Visual Transformers
• SwinCross: Cross-modal Swin Transformer for Head-and-Neck Tumor Segmentation in PET/CT Images
• Deep Learning Technique for Human Parsing: A Survey and Outlook
• WildScenes: A Benchmark for 2D and 3D Semantic Segmentation in Large-scale Natural Environments
• Fully Transformer-Equipped Architecture for End-to-End Referring Video Object Segmentation
• FBNetV5: Neural Architecture Search for Multiple Tasks in One Run
• Parameter-Efficient Fine-Tuning of Large Pretrained Models for Instance Segmentation Tasks
• Weakly supervised training of universal visual concepts for multi-domain semantic segmentation
• ReCLIP++: Learn to Rectify the Bias of CLIP for Unsupervised Semantic Segmentation
• Continual Panoptic Perception: Towards Multi-modal Incremental Interpretation of Remote Sensing Images
• Connecting Domains and Contrasting Samples: A Ladder for Domain Generalization
• FaPN: Feature-aligned Pyramid Network for Dense Image Prediction