Investigating the fidelity of explainable artificial intelligence methods for applications of convolutional neural networks in geoscience

Abstract

Convolutional neural networks (CNNs) have recently attracted great attention in geoscience due to their ability to capture non-linear system behavior and extract predictive spatiotemporal patterns. Given their black-box nature however, and the importance of prediction explainability, methods of explainable artificial intelligence (XAI) are gaining popularity as a means to explain the CNN decision-making strategy. Here, we establish an intercomparison of some of the most popular XAI methods and investigate their fidelity in explaining CNN decisions for geoscientific applications. Our goal is to raise awareness of the theoretical limitations of these methods and gain insight into the relative strengths and weaknesses to help guide best practices. The considered XAI methods are first applied to an idealized attribution benchmark, where the ground truth of explanation of the network is known a priori, to help objectively assess their performance. Secondly, we apply XAI to a climate-related prediction setting, namely to explain a CNN that is trained to predict the number of atmospheric rivers in daily snapshots of climate simulations. Our results highlight several important issues of XAI methods (e.g., gradient shattering, inability to distinguish the sign of attribution, ignorance to zero input) that have previously been overlooked in our field and, if not considered cautiously, may lead to a distorted picture of the CNN decision-making strategy. We envision that our analysis will motivate further investigation into XAI fidelity and will help towards a cautious implementation of XAI in geoscience, which can lead to further exploitation of CNNs and deep learning for prediction problems.

References in corpus (5)

• Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey
• Neural Network Attribution Methods for Problems in Geoscience: A Novel Synthetic Benchmark Dataset
• Indicator patterns of forced change learned by an artificial neural network
• Interpreting the Predictions of Complex ML Models by Layer-wise Relevance Propagation
• Towards Robust Explanations for Deep Neural Networks

Cited by in corpus (9)

• Neural Network Attribution Methods for Problems in Geoscience: A Novel Synthetic Benchmark Dataset
• Opening the Black-Box: A Systematic Review on Explainable AI in Remote Sensing
• Explainable Artificial Intelligence for Bayesian Neural Networks: Towards trustworthy predictions of ocean dynamics
• Time Series Predictions in Unmonitored Sites: A Survey of Machine Learning Techniques in Water Resources
• Learning Closed-form Equations for Subgrid-scale Closures from High-fidelity Data: Promises and Challenges
• Uncertainty Quantification of Wind Gust Predictions in the Northeast United States: An Evidential Neural Network and Explainable Artificial Intelligence Approach
• EvalAttAI: A Holistic Approach to Evaluating Attribution Maps in Robust and Non-Robust Models
• Solving the enigma: Enhancing faithfulness and comprehensibility in explanations of deep networks
• Survey on AI Ethics: A Socio-technical Perspective