Auto-Panoptic: Cooperative Multi-Component Architecture Search for Panoptic Segmentation

Abstract

Panoptic segmentation is posed as a new popular test-bed for the state-of-the-art holistic scene understanding methods with the requirement of simultaneously segmenting both foreground things and background stuff. The state-of-the-art panoptic segmentation network exhibits high structural complexity in different network components, i.e. backbone, proposal-based foreground branch, segmentation-based background branch, and feature fusion module across branches, which heavily relies on expert knowledge and tedious trials. In this work, we propose an efficient, cooperative and highly automated framework to simultaneously search for all main components including backbone, segmentation branches, and feature fusion module in a unified panoptic segmentation pipeline based on the prevailing one-shot Network Architecture Search (NAS) paradigm. Notably, we extend the common single-task NAS into the multi-component scenario by taking the advantage of the newly proposed intra-modular search space and problem-oriented inter-modular search space, which helps us to obtain an optimal network architecture that not only performs well in both instance segmentation and semantic segmentation tasks but also be aware of the reciprocal relations between foreground things and background stuff classes. To relieve the vast computation burden incurred by applying NAS to complicated network architectures, we present a novel path-priority greedy search policy to find a robust, transferrable architecture with significantly reduced searching overhead. Our searched architecture, namely Auto-Panoptic, achieves the new state-of-the-art on the challenging COCO and ADE20K benchmarks. Moreover, extensive experiments are conducted to demonstrate the effectiveness of path-priority policy and transferability of Auto-Panoptic across different datasets. Codes and models are available at: https://github.com/Jacobew/AutoPanoptic.

NeurIPS2020

References in corpus (14)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• EfficientDet: Scalable and Efficient Object Detection
• SMASH: One-Shot Model Architecture Search through HyperNetworks
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• Neural Optimizer Search with Reinforcement Learning
• DeeperLab: Single-Shot Image Parser
• Learning to Fuse Things and Stuff
• Exploring Randomly Wired Neural Networks for Image Recognition
• Panoptic Segmentation with a Joint Semantic and Instance Segmentation Network
• NAS-FCOS: Fast Neural Architecture Search for Object Detection
• Hit-Detector: Hierarchical Trinity Architecture Search for Object Detection
• SM-NAS: Structural-to-Modular Neural Architecture Search for Object Detection
• BANet: Bidirectional Aggregation Network with Occlusion Handling for Panoptic Segmentation
• SpatialFlow: Bridging All Tasks for Panoptic Segmentation

Cited by in corpus (4)

• A Comprehensive Review of Modern Object Segmentation Approaches
• Scaling Wide Residual Networks for Panoptic Segmentation
• DCNAS: Densely Connected Neural Architecture Search for Semantic Image Segmentation
• Ada-Segment: Automated Multi-loss Adaptation for Panoptic Segmentation