Mix-and-Match Tuning for Self-Supervised Semantic Segmentation

Abstract

Deep convolutional networks for semantic image segmentation typically require large-scale labeled data, e.g. ImageNet and MS COCO, for network pre-training. To reduce annotation efforts, self-supervised semantic segmentation is recently proposed to pre-train a network without any human-provided labels. The key of this new form of learning is to design a proxy task (e.g. image colorization), from which a discriminative loss can be formulated on unlabeled data. Many proxy tasks, however, lack the critical supervision signals that could induce discriminative representation for the target image segmentation task. Thus self-supervision's performance is still far from that of supervised pre-training. In this study, we overcome this limitation by incorporating a "mix-and-match" (M&M) tuning stage in the self-supervision pipeline. The proposed approach is readily pluggable to many self-supervision methods and does not use more annotated samples than the original process. Yet, it is capable of boosting the performance of target image segmentation task to surpass fully-supervised pre-trained counterpart. The improvement is made possible by better harnessing the limited pixel-wise annotations in the target dataset. Specifically, we first introduce the "mix" stage, which sparsely samples and mixes patches from the target set to reflect rich and diverse local patch statistics of target images. A "match" stage then forms a class-wise connected graph, which can be used to derive a strong triplet-based discriminative loss for fine-tuning the network. Our paradigm follows the standard practice in existing self-supervised studies and no extra data or label is required. With the proposed M&M approach, for the first time, a self-supervision method can achieve comparable or even better performance compared to its ImageNet pre-trained counterpart on both PASCAL VOC2012 dataset and CityScapes dataset.

To appear in AAAI 2018 as a spotlight paper. More details at the project page: http://mmlab.ie.cuhk.edu.hk/projects/M%26M/

References in corpus (13)

• The Cityscapes Dataset for Semantic Urban Scene Understanding
• Adversarial Feature Learning
• Unsupervised Visual Representation Learning by Context Prediction
• Unsupervised Learning of Visual Representations using Videos
• Data-dependent Initializations of Convolutional Neural Networks
• Semantic Image Segmentation via Deep Parsing Network
• BoxSup: Exploiting Bounding Boxes to Supervise Convolutional Networks for Semantic Segmentation
• Learning Representations for Automatic Colorization
• Colorization as a Proxy Task for Visual Understanding
• Discriminative Unsupervised Feature Learning with Exemplar Convolutional Neural Networks
• Weakly Supervised Semantic Segmentation using Web-Crawled Videos
• Learning Features by Watching Objects Move
• Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Prediction