Multi-Scale Grouped Prototypes for Interpretable Semantic Segmentation

Abstract

Prototypical part learning is emerging as a promising approach for making semantic segmentation interpretable. The model selects real patches seen during training as prototypes and constructs the dense prediction map based on the similarity between parts of the test image and the prototypes. This improves interpretability since the user can inspect the link between the predicted output and the patterns learned by the model in terms of prototypical information. In this paper, we propose a method for interpretable semantic segmentation that leverages multi-scale image representation for prototypical part learning. First, we introduce a prototype layer that explicitly learns diverse prototypical parts at several scales, leading to multi-scale representations in the prototype activation output. Then, we propose a sparse grouping mechanism that produces multi-scale sparse groups of these scale-specific prototypical parts. This provides a deeper understanding of the interactions between multi-scale object representations while enhancing the interpretability of the segmentation model. The experiments conducted on Pascal VOC, Cityscapes, and ADE20K demonstrate that the proposed method increases model sparsity, improves interpretability over existing prototype-based methods, and narrows the performance gap with the non-interpretable counterpart models. Code is available at github.com/eceo-epfl/ScaleProtoSeg.

Accepted at WACV 2025

References in corpus (10)

• Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
• Deep Multi-modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges
• Unmasking Clever Hans Predictors and Assessing What Machines Really Learn
• CA-Net: Comprehensive Attention Convolutional Neural Networks for Explainable Medical Image Segmentation
• Interpretability of machine learning based prediction models in healthcare
• Towards Interpretable Semantic Segmentation via Gradient-weighted Class Activation Mapping
• Interpretable and Steerable Sequence Learning via Prototypes
• Towards a Collective Agenda on AI for Earth Science Data Analysis
• ProtoPShare: Prototype Sharing for Interpretable Image Classification and Similarity Discovery
• On the Interpretability of Part-Prototype Based Classifiers: A Human Centric Analysis