Causal Unsupervised Semantic Segmentation

Abstract

Unsupervised semantic segmentation aims to achieve high-quality semantic grouping without human-labeled annotations. With the advent of self-supervised pre-training, various frameworks utilize the pre-trained features to train prediction heads for unsupervised dense prediction. However, a significant challenge in this unsupervised setup is determining the appropriate level of clustering required for segmenting concepts. To address it, we propose a novel framework, CAusal Unsupervised Semantic sEgmentation (CAUSE), which leverages insights from causal inference. Specifically, we bridge intervention-oriented approach (i.e., frontdoor adjustment) to define suitable two-step tasks for unsupervised prediction. The first step involves constructing a concept clusterbook as a mediator, which represents possible concept prototypes at different levels of granularity in a discretized form. Then, the mediator establishes an explicit link to the subsequent concept-wise self-supervised learning for pixel-level grouping. Through extensive experiments and analyses on various datasets, we corroborate the effectiveness of CAUSE and achieve state-of-the-art performance in unsupervised semantic segmentation.

arXiv commentcode available: https://github.com/ByungKwanLee/Causal-Unsupervised-Segmentation

References in corpus (6)

• Modularity and community structure in networks
• Ward's Hierarchical Clustering Method: Clustering Criterion and Agglomerative Algorithm
• Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials
• Hard Negative Mixing for Contrastive Learning
• ReCo: Retrieve and Co-segment for Zero-shot Transfer
• Boosting Unsupervised Semantic Segmentation with Principal Mask Proposals