Panoptic Segmentation with a Joint Semantic and Instance Segmentation Network

Abstract

We present a single network method for panoptic segmentation. This method combines the predictions from a jointly trained semantic and instance segmentation network using heuristics. Joint training is the first step towards an end-to-end panoptic segmentation network and is faster and more memory efficient than training and predicting with two networks, as done in previous work. The architecture consists of a ResNet-50 feature extractor shared by the semantic segmentation and instance segmentation branch. For instance segmentation, a Mask R-CNN type of architecture is used, while the semantic segmentation branch is augmented with a Pyramid Pooling Module. Results for this method are submitted to the COCO and Mapillary Joint Recognition Challenge 2018. Our approach achieves a PQ score of 17.6 on the Mapillary Vistas validation set and 27.2 on the COCO test-dev set.

arXiv commentTechnical report

References in corpus (2)

• Path Aggregation Network for Instance Segmentation
• Pixel-level Encoding and Depth Layering for Instance-level Semantic Labeling

Cited by in corpus (21)

• Panoptic Feature Fusion Net: A Novel Instance Segmentation Paradigm for Biomedical and Biological Images
• Panoptic Feature Pyramid Networks
• A Comprehensive Review of Modern Object Segmentation Approaches
• Attention-guided Unified Network for Panoptic Segmentation
• An End-to-End Network for Panoptic Segmentation
• UPSNet: A Unified Panoptic Segmentation Network
• Auto-Panoptic: Cooperative Multi-Component Architecture Search for Panoptic Segmentation
• BANet: Bidirectional Aggregation Network with Occlusion Handling for Panoptic Segmentation
• Learning Instance Occlusion for Panoptic Segmentation
• SpatialFlow: Bridging All Tasks for Panoptic Segmentation
• Bidirectional Graph Reasoning Network for Panoptic Segmentation
• Unifying Training and Inference for Panoptic Segmentation
• SOGNet: Scene Overlap Graph Network for Panoptic Segmentation
• Intra-Batch Supervision for Panoptic Segmentation on High-Resolution Images
• TriangleNet: Edge Prior Augmented Network for Semantic Segmentation through Cross-Task Consistency
• CASNet: Common Attribute Support Network for image instance and panoptic segmentation
• Detecting Reflections by Combining Semantic and Instance Segmentation
• Video Panoptic Segmentation
• Generator evaluator-selector net for panoptic image segmentation and splitting unfamiliar objects into parts
• SE-PSNet: Silhouette-based Enhancement Feature for Panoptic Segmentation Network
• Panoptic Segmentation: A Review