Exact and Consistent Interpretation for Piecewise Linear Neural Networks: A Closed Form Solution

Abstract

Strong intelligent machines powered by deep neural networks are increasingly deployed as black boxes to make decisions in risk-sensitive domains, such as finance and medical. To reduce potential risk and build trust with users, it is critical to interpret how such machines make their decisions. Existing works interpret a pre-trained neural network by analyzing hidden neurons, mimicking pre-trained models or approximating local predictions. However, these methods do not provide a guarantee on the exactness and consistency of their interpretation. In this paper, we propose an elegant closed form solution named $OpenBox$ to compute exact and consistent interpretations for the family of Piecewise Linear Neural Networks (PLNN). The major idea is to first transform a PLNN into a mathematically equivalent set of linear classifiers, then interpret each linear classifier by the features that dominate its prediction. We further apply $OpenBox$ to demonstrate the effectiveness of non-negative and sparse constraints on improving the interpretability of PLNNs. The extensive experiments on both synthetic and real world data sets clearly demonstrate the exactness and consistency of our interpretation.

arXiv commentKDD 2018

References in corpus (20)

• Distilling the Knowledge in a Neural Network
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Axiomatic Attribution for Deep Networks
• Learning Important Features Through Propagating Activation Differences
• European Union regulations on algorithmic decision-making and a "right to explanation"
• Understanding Neural Networks Through Deep Visualization
• Do Deep Nets Really Need to be Deep?
• Understanding Black-box Predictions via Influence Functions
• Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
• On the Number of Linear Regions of Deep Neural Networks
• SmoothGrad: removing noise by adding noise
• Distilling a Neural Network Into a Soft Decision Tree
• Visualizing and Understanding Neural Models in NLP
• The (Un)reliability of saliency methods
• Interpreting Blackbox Models via Model Extraction
• On the number of response regions of deep feed forward networks with piece-wise linear activations
• Distilling Knowledge from Deep Networks with Applications to Healthcare Domain
• Beyond Sparsity: Tree Regularization of Deep Models for Interpretability
• Interpretation of Neural Networks is Fragile
• Interpreting Deep Visual Representations via Network Dissection

Cited by in corpus (3)

• Deep Active Learning for Text Classification with Diverse Interpretations
• Safety verification of Neural-Network-based controllers: a set invariance approach
• The Power of Typed Affine Decision Structures: A Case Study