Masked-attention Mask Transformer for Universal Image Segmentation

Abstract

Image segmentation is about grouping pixels with different semantics, e.g., category or instance membership, where each choice of semantics defines a task. While only the semantics of each task differ, current research focuses on designing specialized architectures for each task. We present Masked-attention Mask Transformer (Mask2Former), a new architecture capable of addressing any image segmentation task (panoptic, instance or semantic). Its key components include masked attention, which extracts localized features by constraining cross-attention within predicted mask regions. In addition to reducing the research effort by at least three times, it outperforms the best specialized architectures by a significant margin on four popular datasets. Most notably, Mask2Former sets a new state-of-the-art for panoptic segmentation (57.8 PQ on COCO), instance segmentation (50.1 AP on COCO) and semantic segmentation (57.7 mIoU on ADE20K).

CVPR 2022. Project page/code/models: https://bowenc0221.github.io/mask2former

References in corpus (9)

• Rethinking Atrous Convolution for Semantic Image Segmentation
• PVT v2: Improved Baselines with Pyramid Vision Transformer
• BEiT: BERT Pre-Training of Image Transformers
• Hierarchical Multi-Scale Attention for Semantic Segmentation
• Hybrid Task Cascade for Instance Segmentation
• Panoptic Feature Pyramid Networks
• Scaling Wide Residual Networks for Panoptic Segmentation
• Boundary IoU: Improving Object-Centric Image Segmentation Evaluation
• Simple Training Strategies and Model Scaling for Object Detection