ShapeMask: Learning to Segment Novel Objects by Refining Shape Priors

Abstract

Instance segmentation aims to detect and segment individual objects in a scene. Most existing methods rely on precise mask annotations of every category. However, it is difficult and costly to segment objects in novel categories because a large number of mask annotations is required. We introduce ShapeMask, which learns the intermediate concept of object shape to address the problem of generalization in instance segmentation to novel categories. ShapeMask starts with a bounding box detection and gradually refines it by first estimating the shape of the detected object through a collection of shape priors. Next, ShapeMask refines the coarse shape into an instance level mask by learning instance embeddings. The shape priors provide a strong cue for object-like prediction, and the instance embeddings model the instance specific appearance information. ShapeMask significantly outperforms the state-of-the-art by 6.4 and 3.8 AP when learning across categories, and obtains competitive performance in the fully supervised setting. It is also robust to inaccurate detections, decreased model capacity, and small training data. Moreover, it runs efficiently with 150ms inference time and trains within 11 hours on TPUs. With a larger backbone model, ShapeMask increases the gap with state-of-the-art to 9.4 and 6.2 AP across categories. Code will be released.

References in corpus (6)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Learning to Segment Object Candidates
• Multiscale Combinatorial Grouping for Image Segmentation and Object Proposal Generation
• Semantic Instance Segmentation with a Discriminative Loss Function
• Cascade R-CNN: Delving into High Quality Object Detection
• MaskLab: Instance Segmentation by Refining Object Detection with Semantic and Direction Features

Cited by in corpus (8)

• OCNet: Object Context Network for Scene Parsing
• Segmentation Transformer: Object-Contextual Representations for Semantic Segmentation
• The Best of Both Modes: Separately Leveraging RGB and Depth for Unseen Object Instance Segmentation
• ACFNet: Attentional Class Feature Network for Semantic Segmentation
• X-Ray: Mechanical Search for an Occluded Object by Minimizing Support of Learned Occupancy Distributions
• PolyTransform: Deep Polygon Transformer for Instance Segmentation
• Oriented Boxes for Accurate Instance Segmentation
• Weak Novel Categories without Tears: A Survey on Weak-Shot Learning