Learning to Fuse Things and Stuff

Abstract

We propose an end-to-end learning approach for panoptic segmentation, a novel task unifying instance (things) and semantic (stuff) segmentation. Our model, TASCNet, uses feature maps from a shared backbone network to predict in a single feed-forward pass both things and stuff segmentations. We explicitly constrain these two output distributions through a global things and stuff binary mask to enforce cross-task consistency. Our proposed unified network is competitive with the state of the art on several benchmarks for panoptic segmentation as well as on the individual semantic and instance segmentation tasks.

References in corpus (8)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Focal Loss for Dense Object Detection
• Learning to Segment Object Candidates
• Training Deep Neural Networks on Noisy Labels with Bootstrapping
• Convolutional Feature Masking for Joint Object and Stuff Segmentation
• Fast Scene Understanding for Autonomous Driving
• MaskLab: Instance Segmentation by Refining Object Detection with Semantic and Direction Features
• Weakly- and Semi-Supervised Panoptic Segmentation

Cited by in corpus (30)

• DetectoRS: Detecting Objects with Recursive Feature Pyramid and Switchable Atrous Convolution
• Bird's-Eye-View Panoptic Segmentation Using Monocular Frontal View Images
• Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation
• Panoptic Feature Pyramid Networks
• MGNet: Monocular Geometric Scene Understanding for Autonomous Driving
• Semantically-Guided Representation Learning for Self-Supervised Monocular Depth
• Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation
• MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers
• Scaling Wide Residual Networks for Panoptic Segmentation
• UPSNet: A Unified Panoptic Segmentation Network
• Auto-Panoptic: Cooperative Multi-Component Architecture Search for Panoptic Segmentation
• Naive-Student: Leveraging Semi-Supervised Learning in Video Sequences for Urban Scene Segmentation
• BANet: Bidirectional Aggregation Network with Occlusion Handling for Panoptic Segmentation
• Fully Convolutional Networks for Panoptic Segmentation
• Learning Instance Occlusion for Panoptic Segmentation
• Spatiotemporal Relationship Reasoning for Pedestrian Intent Prediction
• SpatialFlow: Bridging All Tasks for Panoptic Segmentation
• Bidirectional Graph Reasoning Network for Panoptic Segmentation
• Unifying Training and Inference for Panoptic Segmentation
• ViP-DeepLab: Learning Visual Perception with Depth-aware Video Panoptic Segmentation
• SOGNet: Scene Overlap Graph Network for Panoptic Segmentation
• Intra-Batch Supervision for Panoptic Segmentation on High-Resolution Images
• Neural Outlier Rejection for Self-Supervised Keypoint Learning
• CASNet: Common Attribute Support Network for image instance and panoptic segmentation
• Real-Time Panoptic Segmentation from Dense Detections
• Combinatorial Optimization for Panoptic Segmentation: A Fully Differentiable Approach
• Video Panoptic Segmentation
• Generator evaluator-selector net for panoptic image segmentation and splitting unfamiliar objects into parts
• Pixel Consensus Voting for Panoptic Segmentation
• SE-PSNet: Silhouette-based Enhancement Feature for Panoptic Segmentation Network