Local Concept Embeddings for Analysis of Concept Distributions in Vision DNN Feature Spaces

Abstract

Insights into the learned latent representations are imperative for verifying deep neural networks (DNNs) in critical computer vision (CV) tasks. Therefore, state-of-the-art supervised Concept-based eXplainable Artificial Intelligence (C-XAI) methods associate user-defined concepts like ``car'' each with a single vector in the DNN latent space (concept embedding vector). In the case of concept segmentation, these linearly separate between activation map pixels belonging to a concept and those belonging to background. Existing methods for concept segmentation, however, fall short of capturing implicitly learned sub-concepts (e.g., the DNN might split car into ``proximate car'' and ``distant car''), and overlap of user-defined concepts (e.g., between ``bus'' and ``truck''). In other words, they do not capture the full distribution of concept representatives in latent space. For the first time, this work shows that these simplifications are frequently broken and that distribution information can be particularly useful for understanding DNN-learned notions of sub-concepts, concept confusion, and concept outliers. To allow exploration of learned concept distributions, we propose a novel local concept analysis framework. Instead of optimizing a single global concept vector on the complete dataset, it generates a local concept embedding (LoCE) vector for each individual sample. We use the distribution formed by LoCEs to explore the latent concept distribution by fitting Gaussian mixture models (GMMs), hierarchical clustering, and concept-level information retrieval and outlier detection. Despite its context sensitivity, our method's concept segmentation performance is competitive to global baselines. Analysis results are obtained on three datasets and six diverse vision DNN architectures, including vision transformers (ViTs).

arXiv commentThis is the authors accepted manuscript of the article accepted for publication in the International Journal of Computer Vision (IJCV). The final version will be available via SpringerLink upon publication. To cite this work please refer to the final journal version once published

References in corpus (32)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• A Unified Approach to Interpreting Model Predictions
• UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Striving for Simplicity: The All Convolutional Net
• Unmasking Clever Hans Predictors and Assessing What Machines Really Learn
• A Survey on the Explainability of Supervised Machine Learning
• Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
• This Looks Like That: Deep Learning for Interpretable Image Recognition
• Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directions
• Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)
• A survey of the Vision Transformers and their CNN-Transformer based Variants
• A Decade Survey of Content Based Image Retrieval using Deep Learning
• A Comprehensive Taxonomy for Explainable Artificial Intelligence: A Systematic Survey of Surveys on Methods and Concepts
• Concept Whitening for Interpretable Image Recognition
• RISE: Randomized Input Sampling for Explanation of Black-box Models
• Explainable Artificial Intelligence: a Systematic Review
• Rule Extraction Algorithm for Deep Neural Networks: A Review
• Testing and verification of neural-network-based safety-critical control software: A systematic literature review
• Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety
• Post-hoc Concept Bottleneck Models
• Interpretability Beyond Classification Output: Semantic Bottleneck Networks
• Concept Activation Regions: A Generalized Framework For Concept-Based Explanations
• Evaluating the Stability of Semantic Concept Representations in CNNs for Robust Explainability
• Multi-dimensional concept discovery (MCD): A unifying framework with completeness guarantees
• Expressive Explanations of DNNs by Combining Concept Analysis with ILP
• Look at the Variance! Efficient Black-box Explanations with Sobol-based Sensitivity Analysis
• Verification of Size Invariance in DNN Activations using Concept Embeddings
• Making Sense of Dependence: Efficient Black-box Explanations Using Dependence Measure
• Towards Unsupervised Open World Semantic Segmentation
• Enabling Verification of Deep Neural Networks in Perception Tasks Using Fuzzy Logic and Concept Embeddings
• PURE: Turning Polysemantic Neurons Into Pure Features by Identifying Relevant Circuits