TimeSHAP: Explaining Recurrent Models through Sequence Perturbations

Abstract

Although recurrent neural networks (RNNs) are state-of-the-art in numerous sequential decision-making tasks, there has been little research on explaining their predictions. In this work, we present TimeSHAP, a model-agnostic recurrent explainer that builds upon KernelSHAP and extends it to the sequential domain. TimeSHAP computes feature-, timestep-, and cell-level attributions. As sequences may be arbitrarily long, we further propose a pruning method that is shown to dramatically decrease both its computational cost and the variance of its attributions. We use TimeSHAP to explain the predictions of a real-world bank account takeover fraud detection RNN model, and draw key insights from its explanations: i) the model identifies important features and events aligned with what fraud analysts consider cues for account takeover; ii) positive predicted sequences can be pruned to only 10% of the original length, as older events have residual attribution values; iii) the most recent input event of positive predictions only contributes on average to 41% of the model's score; iv) notably high attribution to client's age, suggesting a potential discriminatory reasoning, later confirmed as higher false positive rates for older clients.

arXiv commentAccepted at KDD 2021

References in corpus (7)

• Learning Important Features Through Propagating Activation Differences
• Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
• Model-Agnostic Interpretability of Machine Learning
• RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism
• Patient2Vec: A Personalized Interpretable Deep Representation of the Longitudinal Electronic Health Record
• Automatic Rule Extraction from Long Short Term Memory Networks
• Interpreting a Recurrent Neural Network's Predictions of ICU Mortality Risk

Cited by in corpus (7)

• Mining the contribution of intensive care clinical course to outcome after traumatic brain injury
• A Comprehensive Survey on Self-Interpretable Neural Networks
• Explainable Reinforcement Learning for Formula One Race Strategy
• A Comprehensive Study of Shapley Value in Data Analytics
• CAFO: Feature-Centric Explanation on Time Series Classification
• HyMaTE: A Hybrid Mamba and Transformer Model for EHR Representation Learning
• Few Labels are all you need: A Weakly Supervised Framework for Appliance Localization in Smart-Meter Series