Panoptic Segmentation: A Review

Abstract

Image segmentation for video analysis plays an essential role in different research fields such as smart city, healthcare, computer vision and geoscience, and remote sensing applications. In this regard, a significant effort has been devoted recently to developing novel segmentation strategies; one of the latest outstanding achievements is panoptic segmentation. The latter has resulted from the fusion of semantic and instance segmentation. Explicitly, panoptic segmentation is currently under study to help gain a more nuanced knowledge of the image scenes for video surveillance, crowd counting, self-autonomous driving, medical image analysis, and a deeper understanding of the scenes in general. To that end, we present in this paper the first comprehensive review of existing panoptic segmentation methods to the best of the authors' knowledge. Accordingly, a well-defined taxonomy of existing panoptic techniques is performed based on the nature of the adopted algorithms, application scenarios, and primary objectives. Moreover, the use of panoptic segmentation for annotating new datasets by pseudo-labeling is discussed. Moving on, ablation studies are carried out to understand the panoptic methods from different perspectives. Moreover, evaluation metrics suitable for panoptic segmentation are discussed, and a comparison of the performance of existing solutions is provided to inform the state-of-the-art and identify their limitations and strengths. Lastly, the current challenges the subject technology faces and the future trends attracting considerable interest in the near future are elaborated, which can be a starting point for the upcoming research studies. The papers provided with code are available at: https://github.com/elharroussomar/Awesome-Panoptic-Segmentation

References in corpus (22)

• PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space
• YOLACT++: Better Real-time Instance Segmentation
• DetectoRS: Detecting Objects with Recursive Feature Pyramid and Switchable Atrous Convolution
• A deep learning framework for quality assessment and restoration in video endoscopy
• Automated Evaluation of Semantic Segmentation Robustness for Autonomous Driving
• Panoptic Segmentation with a Joint Semantic and Instance Segmentation Network
• Scaling Wide Residual Networks for Panoptic Segmentation
• MOPT: Multi-Object Panoptic Tracking
• PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things
• Cross-View Regularization for Domain Adaptive Panoptic Segmentation
• LiDAR-based Panoptic Segmentation via Dynamic Shifting Network
• PanoNet: Real-time Panoptic Segmentation through Position-Sensitive Feature Embedding
• Can we cover navigational perception needs of the visually impaired by panoptic segmentation?
• Single-shot Path Integrated Panoptic Segmentation
• Panoptic Edge Detection
• Unifying Instance and Panoptic Segmentation with Dynamic Rank-1 Convolutions
• Instance and Panoptic Segmentation Using Conditional Convolutions
• CASNet: Common Attribute Support Network for image instance and panoptic segmentation
• A Novel Approach for Robust Multi Human Action Recognition and Summarization based on 3D Convolutional Neural Networks
• Fast Object Classification and Meaningful Data Representation of Segmented Lidar Instances
• PanDA: Panoptic Data Augmentation
• Generator evaluator-selector net for panoptic image segmentation and splitting unfamiliar objects into parts