DatUS^2: Data-driven Unsupervised Semantic Segmentation with Pre-trained Self-supervised Vision Transformer

Abstract

Successive proposals of several self-supervised training schemes continue to emerge, taking one step closer to developing a universal foundation model. In this process, the unsupervised downstream tasks are recognized as one of the evaluation methods to validate the quality of visual features learned with a self-supervised training scheme. However, unsupervised dense semantic segmentation has not been explored as a downstream task, which can utilize and evaluate the quality of semantic information introduced in patch-level feature representations during self-supervised training of a vision transformer. Therefore, this paper proposes a novel data-driven approach for unsupervised semantic segmentation (DatUS^2) as a downstream task. DatUS^2 generates semantically consistent and dense pseudo annotate segmentation masks for the unlabeled image dataset without using any visual-prior or synchronized data. We compare these pseudo-annotated segmentation masks with ground truth masks for evaluating recent self-supervised training schemes to learn shared semantic properties at the patch level and discriminative semantic properties at the segment level. Finally, we evaluate existing state-of-the-art self-supervised training schemes with our proposed downstream task, i.e., DatUS^2. Also, the best version of DatUS^2 outperforms the existing state-of-the-art method for the unsupervised dense semantic segmentation task with 15.02% MiOU and 21.47% Pixel accuracy on the SUIM dataset. It also achieves a competitive level of accuracy for a large-scale and complex dataset, i.e., the COCO dataset.

The manuscript contains 13 pages, 9 figures and 7 tables

References in corpus (16)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Fast unfolding of communities in large networks
• Rethinking Atrous Convolution for Semantic Image Segmentation
• A Simple Framework for Contrastive Learning of Visual Representations
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• DINOv2: Learning Robust Visual Features without Supervision
• iBOT: Image BERT Pre-Training with Online Tokenizer
• Unsupervised Semantic Segmentation by Distilling Feature Correspondences
• Localizing Objects with Self-Supervised Transformers and no Labels
• Vision Models Are More Robust And Fair When Pretrained On Uncurated Images Without Supervision
• Discovering Object Masks with Transformers for Unsupervised Semantic Segmentation
• Mugs: A Multi-Granular Self-Supervised Learning Framework
• Self-Supervised Vision-Based Detection of the Active Speaker as Support for Socially-Aware Language Acquisition
• Invariant Information Clustering for Unsupervised Image Classification and Segmentation
• Unicom: Universal and Compact Representation Learning for Image Retrieval
• Fully Self-Supervised Learning for Semantic Segmentation

Cited by in corpus (1)

• AVTENet: A Human-Cognition-Inspired Audio-Visual Transformer-Based Ensemble Network for Video Deepfake Detection