Causal Intervention for Weakly-Supervised Semantic Segmentation

Abstract

We present a causal inference framework to improve Weakly-Supervised Semantic Segmentation (WSSS). Specifically, we aim to generate better pixel-level pseudo-masks by using only image-level labels -- the most crucial step in WSSS. We attribute the cause of the ambiguous boundaries of pseudo-masks to the confounding context, e.g., the correct image-level classification of "horse" and "person" may be not only due to the recognition of each instance, but also their co-occurrence context, making the model inspection (e.g., CAM) hard to distinguish between the boundaries. Inspired by this, we propose a structural causal model to analyze the causalities among images, contexts, and class labels. Based on it, we develop a new method: Context Adjustment (CONTA), to remove the confounding bias in image-level classification and thus provide better pseudo-masks as ground-truth for the subsequent segmentation model. On PASCAL VOC 2012 and MS-COCO, we show that CONTA boosts various popular WSSS methods to new state-of-the-arts.

arXiv commentAccepted as a NeurIPS 2020 oral paper

References in corpus (11)

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Brain Tumor Segmentation with Deep Neural Networks
• Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials
• A Survey of Learning Causality with Data: Problems and Methods
• On Causal and Anticausal Learning
• Interventional Few-Shot Learning
• Learning Independent Causal Mechanisms
• Self-supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation
• On Multi-Cause Causal Inference with Unobserved Confounding: Counterexamples, Impossibility, and Alternatives
• Counterfactuals uncover the modular structure of deep generative models
• Robustly Disentangled Causal Mechanisms: Validating Deep Representations for Interventional Robustness

Cited by in corpus (7)

• Adaptive Patch Contrast for Weakly Supervised Semantic Segmentation
• Facial Action Unit Detection by Adaptively Constraining Self-Attention and Causally Deconfounding Sample
• Towards Deconfounded Image-Text Matching with Causal Inference
• Causal Inference via Style Transfer for Out-of-distribution Generalisation
• Counterfactual Variable Control for Robust and Interpretable Question Answering
• Causality-inspired Single-source Domain Generalization for Medical Image Segmentation
• The Blessings of Unlabeled Background in Untrimmed Videos