What went wrong and when? Instance-wise Feature Importance for Time-series Models

Abstract

Explanations of time series models are useful for high stakes applications like healthcare but have received little attention in machine learning literature. We propose FIT, a framework that evaluates the importance of observations for a multivariate time-series black-box model by quantifying the shift in the predictive distribution over time. FIT defines the importance of an observation based on its contribution to the distributional shift under a KL-divergence that contrasts the predictive distribution against a counterfactual where the rest of the features are unobserved. We also demonstrate the need to control for time-dependent distribution shifts. We compare with state-of-the-art baselines on simulated and real-world clinical data and demonstrate that our approach is superior in identifying important time points and observations throughout the time series.

References in corpus (17)

• Axiomatic Attribution for Deep Networks
• Learning Important Features Through Propagating Activation Differences
• Explaining NonLinear Classification Decisions with Deep Taylor Decomposition
• A Recurrent Latent Variable Model for Sequential Data
• Sanity Checks for Saliency Maps
• Attention is not Explanation
• Visualizing Deep Neural Network Decisions: Prediction Difference Analysis
• RetainVis: Visual Analytics with Interpretable and Interactive Recurrent Neural Networks on Electronic Medical Records
• What Clinicians Want: Contextualizing Explainable Machine Learning for Clinical End Use
• TSViz: Demystification of Deep Learning Models for Time-Series Analysis
• Exploring Interpretable LSTM Neural Networks over Multi-Variable Data
• Investigating the influence of noise and distractors on the interpretation of neural networks
• Explaining Image Classifiers by Counterfactual Generation
• Clinical Intervention Prediction and Understanding using Deep Networks
• An interpretable LSTM neural network for autoregressive exogenous model
• Interpreting Black Box Models via Hypothesis Testing
• Input-Cell Attention Reduces Vanishing Saliency of Recurrent Neural Networks

Cited by in corpus (3)

• Benchmarking Deep Learning Interpretability in Time Series Predictions
• Explaining Time Series Predictions with Dynamic Masks
• Marginal Contribution Feature Importance -- an Axiomatic Approach for The Natural Case