Explaining the Explainer: A First Theoretical Analysis of LIME

Abstract

Machine learning is used more and more often for sensitive applications, sometimes replacing humans in critical decision-making processes. As such, interpretability of these algorithms is a pressing need. One popular algorithm to provide interpretability is LIME (Local Interpretable Model-Agnostic Explanation). In this paper, we provide the first theoretical analysis of LIME. We derive closed-form expressions for the coefficients of the interpretable model when the function to explain is linear. The good news is that these coefficients are proportional to the gradient of the function to explain: LIME indeed discovers meaningful features. However, our analysis also reveals that poor choices of parameters can lead LIME to miss important features.

Accepted to AISTATS 2020

References in corpus (3)

• Learning Important Features Through Propagating Activation Differences
• How to Explain Individual Classification Decisions
• Not Just a Black Box: Learning Important Features Through Propagating Activation Differences

Cited by in corpus (5)

• Explainable $k$-Means and $k$-Medians Clustering
• LIMEtree: Consistent and Faithful Surrogate Explanations of Multiple Classes
• SurvLIME-Inf: A simplified modification of SurvLIME for explanation of machine learning survival models
• Connecting Interpretability and Robustness in Decision Trees through Separation
• Ensembles of Random SHAPs