Explaining Bayesian Neural Networks
arXiv:2108.10346
Abstract
To advance the transparency of learning machines such as Deep Neural Networks (DNNs), the field of Explainable AI (XAI) was established to provide interpretations of DNNs' predictions. While different explanation techniques exist, a popular approach is given in the form of attribution maps, which illustrate, given a particular data point, the relevant patterns the model has used for making its prediction. Although Bayesian models such as Bayesian Neural Networks (BNNs) have a limited form of transparency built-in through their prior weight distribution, they lack explanations of their predictions for given instances. In this work, we take a step toward combining these two perspectives by examining how local attributions can be extended to BNNs. Within the Bayesian framework, network weights follow a probability distribution; hence, the standard point explanation extends naturally to an explanation distribution. Viewing explanations probabilistically, we aggregate and analyze multiple local attributions drawn from an approximate posterior to explore variability in explanation patterns. The diversity of explanations offers a way to further explore how predictive rationales may vary across posterior samples. Quantitative and qualitative experiments on toy and benchmark data, as well as on a real-world pathology dataset, illustrate that our framework enriches standard explanations with uncertainty information and may support the visualization of explanation stability.
25 pages, 8 figures Accepted to Transactions on Machine Learning Research
References in corpus (30)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- Language Models are Few-Shot Learners
- Axiomatic Attribution for Deep Networks
- Methods for Interpreting and Understanding Deep Neural Networks
- A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAI
- Understanding Neural Networks Through Deep Visualization
- Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
- Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- SmoothGrad: removing noise by adding noise
- How to Explain Individual Classification Decisions
- StarCraft II: A New Challenge for Reinforcement Learning
- Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey
- Variational Dropout and the Local Reparameterization Trick
- Variational Dropout Sparsifies Deep Neural Networks
- Learning how to explain neural networks: PatternNet and PatternAttribution
- "What is Relevant in a Text Document?": An Interpretable Machine Learning Approach
- Go-Explore: a New Approach for Hard-Exploration Problems
- Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networks
- Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding
- Practical Deep Learning with Bayesian Principles
- Fooling Neural Network Interpretations via Adversarial Model Manipulation
- Understanding Back-Translation at Scale
- The Case for Bayesian Deep Learning
- Fixing the train-test resolution discrepancy: FixEfficientNet
- Feature Importance Measure for Non-linear Learning Algorithms
- Estimating Model Uncertainty of Neural Networks in Sparse Information Form
- Bayesian Optimization Meets Laplace Approximation for Robotic Introspection