Bridging Category-level and Instance-level Semantic Image Segmentation

Abstract

We propose an approach to instance-level image segmentation that is built on top of category-level segmentation. Specifically, for each pixel in a semantic category mask, its corresponding instance bounding box is predicted using a deep fully convolutional regression network. Thus it follows a different pipeline to the popular detect-then-segment approaches that first predict instances' bounding boxes, which are the current state-of-the-art in instance segmentation. We show that, by leveraging the strength of our state-of-the-art semantic segmentation models, the proposed method can achieve comparable or even better results to detect-then-segment approaches. We make the following contributions. (i) First, we propose a simple yet effective approach to semantic instance segmentation. (ii) Second, we propose an online bootstrapping method during training, which is critically important for achieving good performance for both semantic category segmentation and instance-level segmentation. (iii) As the performance of semantic category segmentation has a significant impact on the instance-level segmentation, which is the second step of our approach, we train fully convolutional residual networks to achieve the best semantic category segmentation accuracy. On the PASCAL VOC 2012 dataset, we obtain the currently best mean intersection-over-union score of 79.1%. (iv) We also achieve state-of-the-art results for instance-level segmentation.

arXiv comment14 pages. arXiv admin note: substantial text overlap with arXiv:1604.04339

References in corpus (9)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials
• Training Very Deep Networks
• Identity Mappings in Deep Residual Networks
• Online Batch Selection for Faster Training of Neural Networks
• Proposal-free Network for Instance-level Object Segmentation
• Pixel-level Encoding and Depth Layering for Instance-level Semantic Labeling
• Exploring Context with Deep Structured models for Semantic Segmentation

Cited by in corpus (43)

• DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs
• Evolution of Image Segmentation using Deep Convolutional Neural Network: A Survey
• Pyramid Scene Parsing Network
• Deep learning to achieve clinically applicable segmentation of head and neck anatomy for radiotherapy
• Wider or Deeper: Revisiting the ResNet Model for Visual Recognition
• Hybrid Task Cascade for Instance Segmentation
• DeeperLab: Single-Shot Image Parser
• Context Encoding for Semantic Segmentation
• Real-time Semantic Image Segmentation via Spatial Sparsity
• Do We Really Need Dice? The Hidden Region-Size Biases of Segmentation Losses
• Estimating Depth from Monocular Images as Classification Using Deep Fully Convolutional Residual Networks
• Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation
• MaskLab: Instance Segmentation by Refining Object Detection with Semantic and Direction Features
• SPGNet: Semantic Prediction Guidance for Scene Parsing
• FEELVOS: Fast End-to-End Embedding Learning for Video Object Segmentation
• Segmentation Loss Odyssey
• A Hierarchical Probabilistic U-Net for Modeling Multi-Scale Ambiguities
• Iteratively Trained Interactive Segmentation
• Automatic Analysis of Sewer Pipes Based on Unrolled Monocular Fisheye Images
• InstanceCut: from Edges to Instances with MultiCut
• ORDNet: Capturing Omni-Range Dependencies for Scene Parsing
• Improving Fully Convolution Network for Semantic Segmentation
• Full-Resolution Residual Networks for Semantic Segmentation in Street Scenes
• Correlation Maximized Structural Similarity Loss for Semantic Segmentation
• Auto Seg-Loss: Searching Metric Surrogates for Semantic Segmentation
• Tensor Low-Rank Reconstruction for Semantic Segmentation
• Semantic Correlation Promoted Shape-Variant Context for Segmentation
• SpatialFlow: Bridging All Tasks for Panoptic Segmentation
• Learning Statistical Texture for Semantic Segmentation
• Decoders Matter for Semantic Segmentation: Data-Dependent Decoding Enables Flexible Feature Aggregation
• Single-shot Path Integrated Panoptic Segmentation
• Dilated SpineNet for Semantic Segmentation
• Beyond Forward Shortcuts: Fully Convolutional Master-Slave Networks (MSNets) with Backward Skip Connections for Semantic Segmentation
• AutoLoss: Automated Loss Function Search in Recommendations
• MUNet: Motion Uncertainty-aware Semi-supervised Video Object Segmentation
• Distance Guided Channel Weighting for Semantic Segmentation
• AutoLoss-Zero: Searching Loss Functions from Scratch for Generic Tasks
• Loss Ensembles for Extremely Imbalanced Segmentation
• Real-Time Panoptic Segmentation from Dense Detections
• Mass Displacement Networks
• Inspect Transfer Learning Architecture with Dilated Convolution
• Bounding Box Embedding for Single Shot Person Instance Segmentation
• Meta R-CNN : Towards General Solver for Instance-level Few-shot Learning