Weakly- and Semi-Supervised Panoptic Segmentation

Abstract

We present a weakly supervised model that jointly performs both semantic- and instance-segmentation -- a particularly relevant problem given the substantial cost of obtaining pixel-perfect annotation for these tasks. In contrast to many popular instance segmentation approaches based on object detectors, our method does not predict any overlapping instances. Moreover, we are able to segment both "thing" and "stuff" classes, and thus explain all the pixels in the image. "Thing" classes are weakly-supervised with bounding boxes, and "stuff" with image-level tags. We obtain state-of-the-art results on Pascal VOC, for both full and weak supervision (which achieves about 95% of fully-supervised performance). Furthermore, we present the first weakly-supervised results on Cityscapes for both semantic- and instance-segmentation. Finally, we use our weakly supervised framework to analyse the relationship between annotation quality and predictive performance, which is of interest to dataset creators.

arXiv commentECCV 2018. The first two authors contributed equally

References in corpus (17)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Rethinking Atrous Convolution for Semantic Image Segmentation
• Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials
• The Cityscapes Dataset for Semantic Urban Scene Understanding
• STC: A Simple to Complex Framework for Weakly-supervised Semantic Segmentation
• Weakly- and Semi-Supervised Learning of a DCNN for Semantic Image Segmentation
• Path Aggregation Network for Instance Segmentation
• Revisiting Unreasonable Effectiveness of Data in Deep Learning Era
• Simultaneous Detection and Segmentation
• BoxSup: Exploiting Bounding Boxes to Supervise Convolutional Networks for Semantic Segmentation
• Proposal-free Network for Instance-level Object Segmentation
• PixelNet: Representation of the pixels, by the pixels, and for the pixels
• Pixel-level Encoding and Depth Layering for Instance-level Semantic Labeling
• Monocular Object Instance Segmentation and Depth Ordering with CNNs
• Exploiting saliency for object segmentation from image level labels
• Learning to Segment Every Thing
• From Image-level to Pixel-level Labeling with Convolutional Networks

Cited by in corpus (2)

• Learning to Fuse Things and Stuff
• Class Activation Map Generation by Representative Class Selection and Multi-Layer Feature Fusion