MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers

Abstract

We present MaX-DeepLab, the first end-to-end model for panoptic segmentation. Our approach simplifies the current pipeline that depends heavily on surrogate sub-tasks and hand-designed components, such as box detection, non-maximum suppression, thing-stuff merging, etc. Although these sub-tasks are tackled by area experts, they fail to comprehensively solve the target task. By contrast, our MaX-DeepLab directly predicts class-labeled masks with a mask transformer, and is trained with a panoptic quality inspired loss via bipartite matching. Our mask transformer employs a dual-path architecture that introduces a global memory path in addition to a CNN path, allowing direct communication with any CNN layers. As a result, MaX-DeepLab shows a significant 7.1% PQ gain in the box-free regime on the challenging COCO dataset, closing the gap between box-based and box-free methods for the first time. A small variant of MaX-DeepLab improves 3.0% PQ over DETR with similar parameters and M-Adds. Furthermore, MaX-DeepLab, without test time augmentation, achieves new state-of-the-art 51.3% PQ on COCO test-dev set. Code is available at https://github.com/google-research/deeplab2.

arXiv commentCVPR 2021

References in corpus (14)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Rethinking Atrous Convolution for Semantic Image Segmentation
• Longformer: The Long-Document Transformer
• Deformable DETR: Deformable Transformers for End-to-End Object Detection
• Linformer: Self-Attention with Linear Complexity
• Generating Long Sequences with Sparse Transformers
• Axial Attention in Multidimensional Transformers
• Reformer: The Efficient Transformer
• nnU-Net: Self-adapting Framework for U-Net-Based Medical Image Segmentation
• CondConv: Conditionally Parameterized Convolutions for Efficient Inference
• DeeperLab: Single-Shot Image Parser
• Learning to Fuse Things and Stuff
• GMAT: Global Memory Augmentation for Transformers

Cited by in corpus (6)

• K-Net: Towards Unified Image Segmentation
• TFPose: Direct Human Pose Estimation with Transformers
• Vision Transformer Pruning
• Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling
• Local-to-Global Self-Attention in Vision Transformers
• A State-of-the-art Survey of Artificial Neural Networks for Whole-slide Image Analysis:from Popular Convolutional Neural Networks to Potential Visual Transformers