Captum: A unified and generic model interpretability library for PyTorch

Abstract

In this paper we introduce a novel, unified, open-source model interpretability library for PyTorch [12]. The library contains generic implementations of a number of gradient and perturbation-based attribution algorithms, also known as feature, neuron and layer importance algorithms, as well as a set of evaluation metrics for these algorithms. It can be used for both classification and non-classification models including graph-structured models built on Neural Networks (NN). In this paper we give a high-level overview of supported attribution algorithms and show how to perform memory-efficient and scalable computations. We emphasize that the three main characteristics of the library are multimodality, extensibility and ease of use. Multimodality supports different modality of inputs such as image, text, audio or video. Extensibility allows adding new algorithms and features. The library is also designed for easy understanding and use. Besides, we also introduce an interactive visualization tool called Captum Insights that is built on top of Captum library and allows sample-based model debugging and visualization using feature importance metrics.

References in corpus (6)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Axiomatic Attribution for Deep Networks
• Learning Important Features Through Propagating Activation Differences
• SmoothGrad: removing noise by adding noise
• Sanity Checks for Saliency Maps
• Benchmarking Attribution Methods with Relative Feature Importance

Cited by in corpus (55)

• Dermatologist-like explainable AI enhances trust and confidence in diagnosing melanoma
• Evaluating explainable artificial intelligence methods for multi-label deep learning classification tasks in remote sensing
• Evaluation of post-hoc interpretability methods in time-series classification
• Continuous Human Action Recognition for Human-Machine Interaction: A Review
• Fairness via Explanation Quality: Evaluating Disparities in the Quality of Post hoc Explanations
• Toward Accurate Interpretable Predictions of Materials Properties within Transformer Language Models
• Hierarchical Disentanglement-Alignment Network for Robust SAR Vehicle Recognition
• Inseq: An Interpretability Toolkit for Sequence Generation Models
• ferret: a Framework for Benchmarking Explainers on Transformers
• Gifsplanation via Latent Shift: A Simple Autoencoder Approach to Counterfactual Generation for Chest X-rays
• Do Concept Bottleneck Models Learn as Intended?
• Disentangled Explanations of Neural Network Predictions by Finding Relevant Subspaces
• Anatomically aware dual-hop learning for pulmonary embolism detection in CT pulmonary angiograms
• Saliency is a Possible Red Herring When Diagnosing Poor Generalization
• TRUST-LAPSE: An Explainable and Actionable Mistrust Scoring Framework for Model Monitoring
• An Open API Architecture to Discover the Trustworthy Explanation of Cloud AI Services
• Multilingual Multi-Aspect Explainability Analyses on Machine Reading Comprehension Models
• Speaker-Conditioned Hierarchical Modeling for Automated Speech Scoring
• A Graph Based Neural Network Approach to Immune Profiling of Multiplexed Tissue Samples
• Does Your Model Think Like an Engineer? Explainable AI for Bearing Fault Detection with Deep Learning
• Robustness of Visual Explanations to Common Data Augmentation
• EvalAttAI: A Holistic Approach to Evaluating Attribution Maps in Robust and Non-Robust Models
• Robust Models Are More Interpretable Because Attributions Look Normal
• Generalizability Analysis of Graph-based Trajectory Predictor with Vectorized Representation
• The Thousand Faces of Explainable AI Along the Machine Learning Life Cycle: Industrial Reality and Current State of Research
• VendorLink: An NLP approach for Identifying & Linking Vendor Migrants & Potential Aliases on Darknet Markets
• Bridging the Gap: Gaze Events as Interpretable Concepts to Explain Deep Neural Sequence Models
• PhenoLinker: Phenotype-Gene Link Prediction and Explanation using Heterogeneous Graph Neural Networks
• Massive $\nu$s through the CNN lens: interpreting the field-level neutrino mass information in weak lensing
• Quantitative Analysis of Primary Attribution Explainable Artificial Intelligence Methods for Remote Sensing Image Classification
• Visualizing Automatic Speech Recognition -- Means for a Better Understanding?
• Challenges for cognitive decoding using deep learning methods
• Good-looking but Lacking Faithfulness: Understanding Local Explanation Methods through Trend-based Testing
• Scene Text Recognition Models Explainability Using Local Features
• AutoNLU: Detecting, root-causing, and fixing NLU model errors
• Bangla Text Classification using Transformers
• Example-Based Concept Analysis Framework for Deep Weather Forecast Models
• Model Interpretability and Rationale Extraction by Input Mask Optimization
• Studying Limits of Explainability by Integrated Gradients for Gene Expression Models
• Quantifying Explainability in NLP and Analyzing Algorithms for Performance-Explainability Tradeoff
• Inferring halo masses with Graph Neural Networks
• Explaining Deep Neural Networks for Bearing Fault Detection with Vibration Concepts
• Minimal Sufficient Views: A DNN model making predictions with more evidence has higher accuracy
• FW-Shapley: Real-time Estimation of Weighted Shapley Values
• PSF-based Analysis for Detecting Unresolved Wide Binaries
• Deep Learning-Based Prediction of High Explosive Induced Fluid Dynamics
• A Review of Bangla Natural Language Processing Tasks and the Utility of Transformer Models
• GECOBench: A Gender-Controlled Text Dataset and Benchmark for Quantifying Biases in Explanations
• Human-Imitating Metrics for Training and Evaluating Privacy Preserving Emotion Recognition Models Using Sociolinguistic Knowledge
• Multi-concept adversarial attacks
• Explaining Classes through Word Attribution
• Explainability Requires Interactivity
• Attribution Mask: Filtering Out Irrelevant Features By Recursively Focusing Attention on Inputs of DNNs
• Picasso: Model-free Feature Visualization
• A Robust Unsupervised Ensemble of Feature-Based Explanations using Restricted Boltzmann Machines