K-Net: Towards Unified Image Segmentation

Abstract

Semantic, instance, and panoptic segmentations have been addressed using different and specialized frameworks despite their underlying connections. This paper presents a unified, simple, and effective framework for these essentially similar tasks. The framework, named K-Net, segments both instances and semantic categories consistently by a group of learnable kernels, where each kernel is responsible for generating a mask for either a potential instance or a stuff class. To remedy the difficulties of distinguishing various instances, we propose a kernel update strategy that enables each kernel dynamic and conditional on its meaningful group in the input image. K-Net can be trained in an end-to-end manner with bipartite matching, and its training and inference are naturally NMS-free and box-free. Without bells and whistles, K-Net surpasses all previous published state-of-the-art single-model results of panoptic segmentation on MS COCO test-dev split and semantic segmentation on ADE20K val split with 55.2% PQ and 54.3% mIoU, respectively. Its instance segmentation performance is also on par with Cascade Mask R-CNN on MS COCO with 60%-90% faster inference speeds. Code and models will be released at https://github.com/ZwwWayne/K-Net/.

Camera ready for NeurIPS2021

References in corpus (14)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Decoupled Weight Decay Regularization
• Rethinking Atrous Convolution for Semantic Image Segmentation
• Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
• The Cityscapes Dataset for Semantic Urban Scene Understanding
• MMDetection: Open MMLab Detection Toolbox and Benchmark
• Associative Embedding: End-to-End Learning for Joint Detection and Grouping
• Hybrid Task Cascade for Instance Segmentation
• DeeperLab: Single-Shot Image Parser
• Context Encoding for Semantic Segmentation
• Expectation-Maximization Attention Networks for Semantic Segmentation
• Panoptic Feature Pyramid Networks
• TensorMask: A Foundation for Dense Object Segmentation
• MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers

Cited by in corpus (3)

• A Comprehensive Review of Modern Object Segmentation Approaches
• RT-K-Net: Revisiting K-Net for Real-Time Panoptic Segmentation
• Intra-Batch Supervision for Panoptic Segmentation on High-Resolution Images