SpatialFlow: Bridging All Tasks for Panoptic Segmentation

Abstract

Object location is fundamental to panoptic segmentation as it is related to all things and stuff in the image scene. Knowing the locations of objects in the image provides clues for segmenting and helps the network better understand the scene. How to integrate object location in both thing and stuff segmentation is a crucial problem. In this paper, we propose spatial information flows to achieve this objective. The flows can bridge all sub-tasks in panoptic segmentation by delivering the object's spatial context from the box regression task to others. More importantly, we design four parallel sub-networks to get a preferable adaptation of object spatial information in sub-tasks. Upon the sub-networks and the flows, we present a location-aware and unified framework for panoptic segmentation, denoted as SpatialFlow. We perform a detailed ablation study on each component and conduct extensive experiments to prove the effectiveness of SpatialFlow. Furthermore, we achieve state-of-the-art results, which are $47.9$ PQ and $62.5$ PQ respectively on MS-COCO and Cityscapes panoptic benchmarks. Code will be available at https://github.com/chensnathan/SpatialFlow.

arXiv commentAccepted to IEEE TCSVT

References in corpus (15)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Rethinking Atrous Convolution for Semantic Image Segmentation
• R-FCN: Object Detection via Region-based Fully Convolutional Networks
• MMDetection: Open MMLab Detection Toolbox and Benchmark
• Hybrid Task Cascade for Instance Segmentation
• DeeperLab: Single-Shot Image Parser
• Bridging Category-level and Instance-level Semantic Image Segmentation
• RetinaMask: Learning to predict masks improves state-of-the-art single-shot detection for free
• Learning to Fuse Things and Stuff
• Panoptic Segmentation with a Joint Semantic and Instance Segmentation Network
• Panoptic Feature Pyramid Networks
• Attention-guided Unified Network for Panoptic Segmentation
• An End-to-End Network for Panoptic Segmentation
• UPSNet: A Unified Panoptic Segmentation Network

Cited by in corpus (6)

• DetectoRS: Detecting Objects with Recursive Feature Pyramid and Switchable Atrous Convolution
• Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation
• Scaling Wide Residual Networks for Panoptic Segmentation
• Auto-Panoptic: Cooperative Multi-Component Architecture Search for Panoptic Segmentation
• Bidirectional Graph Reasoning Network for Panoptic Segmentation
• Ada-Segment: Automated Multi-loss Adaptation for Panoptic Segmentation