Exact and Consistent Interpretation of Piecewise Linear Models Hidden behind APIs: A Closed Form Solution

Abstract

More and more AI services are provided through APIs on cloud where predictive models are hidden behind APIs. To build trust with users and reduce potential application risk, it is important to interpret how such predictive models hidden behind APIs make their decisions. The biggest challenge of interpreting such predictions is that no access to model parameters or training data is available. Existing works interpret the predictions of a model hidden behind an API by heuristically probing the response of the API with perturbed input instances. However, these methods do not provide any guarantee on the exactness and consistency of their interpretations. In this paper, we propose an elegant closed form solution named OpenAPI to compute exact and consistent interpretations for the family of Piecewise Linear Models (PLM), which includes many popular classification models. The major idea is to first construct a set of overdetermined linear equation systems with a small set of perturbed instances and the predictions made by the model on those instances. Then, we solve the equation systems to identify the decision features that are responsible for the prediction on an input instance. Our extensive experiments clearly demonstrate the exactness and consistency of our method.

References in corpus (18)

• Distilling the Knowledge in a Neural Network
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Axiomatic Attribution for Deep Networks
• Learning Important Features Through Propagating Activation Differences
• European Union regulations on algorithmic decision-making and a "right to explanation"
• ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Models
• Do Deep Nets Really Need to be Deep?
• Understanding Black-box Predictions via Influence Functions
• On the Number of Linear Regions of Deep Neural Networks
• SmoothGrad: removing noise by adding noise
• Not Just a Black Box: Learning Important Features Through Propagating Activation Differences
• Distilling a Neural Network Into a Soft Decision Tree
• Prototype selection for interpretable classification
• On the number of response regions of deep feed forward networks with piece-wise linear activations
• Distilling Knowledge from Deep Networks with Applications to Healthcare Domain
• Beyond Sparsity: Tree Regularization of Deep Models for Interpretability
• Explaining Deep Learning Models - A Bayesian Non-parametric Approach
• "Influence Sketching": Finding Influential Samples In Large-Scale Regressions