DeeperLab: Single-Shot Image Parser

Abstract

We present a single-shot, bottom-up approach for whole image parsing. Whole image parsing, also known as Panoptic Segmentation, generalizes the tasks of semantic segmentation for 'stuff' classes and instance segmentation for 'thing' classes, assigning both semantic and instance labels to every pixel in an image. Recent approaches to whole image parsing typically employ separate standalone modules for the constituent semantic and instance segmentation tasks and require multiple passes of inference. Instead, the proposed DeeperLab image parser performs whole image parsing with a significantly simpler, fully convolutional approach that jointly addresses the semantic and instance segmentation tasks in a single-shot manner, resulting in a streamlined system that better lends itself to fast processing. For quantitative evaluation, we use both the instance-based Panoptic Quality (PQ) metric and the proposed region-based Parsing Covering (PC) metric, which better captures the image parsing quality on 'stuff' classes and larger object instances. We report experimental results on the challenging Mapillary Vistas dataset, in which our single model achieves 31.95% (val) / 31.6% PQ (test) and 55.26% PC (val) with 3 frames per second (fps) on GPU or near real-time speed (22.6 fps on GPU) with reduced accuracy.

20 pages. The code of the proposed Parsing Covering metric is available at http://deeperlab.mit.edu

References in corpus (16)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Rethinking Atrous Convolution for Semantic Image Segmentation
• R-FCN: Object Detection via Region-based Fully Convolutional Networks
• ParseNet: Looking Wider to See Better
• Learning to Segment Object Candidates
• Associative Embedding: End-to-End Learning for Joint Detection and Grouping
• Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics
• Convolutional Feature Masking for Joint Object and Stuff Segmentation
• Semantic Instance Segmentation with a Discriminative Loss Function
• Semantic Instance Segmentation via Deep Metric Learning
• Bridging Category-level and Instance-level Semantic Image Segmentation
• Proposal-free Network for Instance-level Object Segmentation
• Stacked Deconvolutional Network for Semantic Segmentation
• Pixel-level Encoding and Depth Layering for Instance-level Semantic Labeling

Cited by in corpus (45)

• Hierarchical Multi-Scale Attention for Semantic Segmentation
• K-Net: Towards Unified Image Segmentation
• DetectoRS: Detecting Objects with Recursive Feature Pyramid and Switchable Atrous Convolution
• HigherHRNet: Scale-Aware Representation Learning for Bottom-Up Human Pose Estimation
• Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation
• Interlaced Sparse Self-Attention for Semantic Segmentation
• MGNet: Monocular Geometric Scene Understanding for Autonomous Driving
• Efficient Multi-Task RGB-D Scene Analysis for Indoor Environments
• A Comprehensive Review of Modern Object Segmentation Approaches
• Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation
• SPGNet: Semantic Prediction Guidance for Scene Parsing
• DeepLab2: A TensorFlow Library for Deep Labeling
• MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers
• SSAP: Single-Shot Instance Segmentation With Affinity Pyramid
• Scaling Wide Residual Networks for Panoptic Segmentation
• Efficient Multi-Task Scene Analysis with RGB-D Transformers
• Auto-Panoptic: Cooperative Multi-Component Architecture Search for Panoptic Segmentation
• Naive-Student: Leveraging Semi-Supervised Learning in Video Sequences for Urban Scene Segmentation
• Fully Convolutional Networks for Panoptic Segmentation
• Deep Affinity Net: Instance Segmentation via Affinity
• Panoptic-PolarNet: Proposal-free LiDAR Point Cloud Panoptic Segmentation
• SpatialFlow: Bridging All Tasks for Panoptic Segmentation
• SegSort: Segmentation by Discriminative Sorting of Segments
• PanoNet: Real-time Panoptic Segmentation through Position-Sensitive Feature Embedding
• STEP: Segmenting and Tracking Every Pixel
• RT-K-Net: Revisiting K-Net for Real-Time Panoptic Segmentation
• Single-shot Path Integrated Panoptic Segmentation
• Unifying Training and Inference for Panoptic Segmentation
• ViP-DeepLab: Learning Visual Perception with Depth-aware Video Panoptic Segmentation
• SOGNet: Scene Overlap Graph Network for Panoptic Segmentation
• Intra-Batch Supervision for Panoptic Segmentation on High-Resolution Images
• Instance and Panoptic Segmentation Using Conditional Convolutions
• Exemplar-Based Open-Set Panoptic Segmentation Network
• Panoster: End-to-end Panoptic Segmentation of LiDAR Point Clouds
• CPSeg: Cluster-free Panoptic Segmentation of 3D LiDAR Point Clouds
• Rethinking Text Segmentation: A Novel Dataset and A Text-Specific Refinement Approach
• Real-Time Panoptic Segmentation from Dense Detections
• CASNet: Common Attribute Support Network for image instance and panoptic segmentation
• Combinatorial Optimization for Panoptic Segmentation: A Fully Differentiable Approach
• Video Panoptic Segmentation
• SpineOne: A One-Stage Detection Framework for Degenerative Discs and Vertebrae
• Dense Prediction with Attentive Feature Aggregation
• Video Instance Segmentation by Instance Flow Assembly
• Pixel Consensus Voting for Panoptic Segmentation
• SE-PSNet: Silhouette-based Enhancement Feature for Panoptic Segmentation Network