FovEx: Human-Inspired Explanations for Vision Transformers and Convolutional Neural Networks

Abstract

Explainability in artificial intelligence (XAI) remains a crucial aspect for fostering trust and understanding in machine learning models. Current visual explanation techniques, such as gradient-based or class-activation-based methods, often exhibit a strong dependence on specific model architectures. Conversely, perturbation-based methods, despite being model-agnostic, are computationally expensive as they require evaluating models on a large number of forward passes. In this work, we introduce Foveation-based Explanations (FovEx), a novel XAI method inspired by human vision. FovEx seamlessly integrates biologically inspired perturbations by iteratively creating foveated renderings of the image and combines them with gradient-based visual explorations to determine locations of interest efficiently. These locations are selected to maximize the performance of the model to be explained with respect to the downstream task and then combined to generate an attribution map. We provide a thorough evaluation with qualitative and quantitative assessments on established benchmarks. Our method achieves state-of-the-art performance on both transformers (on 4 out of 5 metrics) and convolutional models (on 3 out of 5 metrics), demonstrating its versatility among various architectures. Furthermore, we show the alignment between the explanation map produced by FovEx and human gaze patterns (+14\% in NSS compared to RISE, +203\% in NSS compared to GradCAM). This comparison enhances our confidence in FovEx's ability to close the interpretation gap between humans and machines.

arXiv commentAccepted in the International Journal of Computer Vision (Springer Nature)

References in corpus (13)

• SSD: Single Shot MultiBox Detector
• Learning Transferable Visual Models From Natural Language Supervision
• Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks
• European Union regulations on algorithmic decision-making and a "right to explanation"
• Interpretable Explanations of Black Boxes by Meaningful Perturbation
• Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
• Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directions
• RISE: Randomized Input Sampling for Explanation of Black-box Models
• From Attribution Maps to Human-Understandable Explanations through Concept Relevance Propagation
• Deep Transfer Learning for Industrial Automation: A Review and Discussion of New Techniques for Data-Driven Machine Learning
• Robust Explainability: A Tutorial on Gradient-Based Attribution Methods for Deep Neural Networks
• Analyzing Vision Transformers for Image Classification in Class Embedding Space
• Contrastive Language-Image Pretrained Models are Zero-Shot Human Scanpath Predictors