Sparse Oblique Decision Trees: A Tool to Understand and Manipulate Neural Net Features

Abstract

The widespread deployment of deep nets in practical applications has lead to a growing desire to understand how and why such black-box methods perform prediction. Much work has focused on understanding what part of the input pattern (an image, say) is responsible for a particular class being predicted, and how the input may be manipulated to predict a different class. We focus instead on understanding which of the internal features computed by the neural net are responsible for a particular class. We achieve this by mimicking part of the neural net with an oblique decision tree having sparse weight vectors at the decision nodes. Using the recently proposed Tree Alternating Optimization (TAO) algorithm, we are able to learn trees that are both highly accurate and interpretable. Such trees can faithfully mimic the part of the neural net they replaced, and hence they can provide insights into the deep net black box. Further, we show we can easily manipulate the neural net features in order to make the net predict, or not predict, a given class, thus showing that it is possible to carry out adversarial attacks at the level of the features. These insights and manipulations apply globally to the entire training and test set, not just at a local (single-instance) level. We demonstrate this robustly in the MNIST and ImageNet datasets with LeNet5 and VGG networks.

arXiv commentAppears in Data Mining and Knowledge Discovery (2023), Special Issue on Explainable and Interpretable Machine Learning and Data Mining

References in corpus (18)

• Scikit-learn: Machine Learning in Python
• XGBoost: A Scalable Tree Boosting System
• Axiomatic Attribution for Deep Networks
• Methods for Interpreting and Understanding Deep Neural Networks
• Learning Important Features Through Propagating Activation Differences
• Confounding variables can degrade generalization performance of radiological deep learning models
• Explaining NonLinear Classification Decisions with Deep Taylor Decomposition
• Understanding Black-box Predictions via Influence Functions
• Interpretable Explanations of Black Boxes by Meaningful Perturbation
• Sanity Checks for Saliency Maps
• Visualizing Deep Convolutional Neural Networks Using Natural Pre-Images
• Estimating Training Data Influence by Tracing Gradient Descent
• Hierarchical interpretations for neural network predictions
• Representer Point Selection for Explaining Deep Neural Networks
• Understanding Intra-Class Knowledge Inside CNN
• Visualizing Deep Networks by Optimizing with Integrated Gradients
• An Experimental Comparison of Old and New Decision Tree Algorithms
• Sampling the "Inverse Set" of a Neuron: An Approach to Understanding Neural Nets