NBDT: Neural-Backed Decision Trees

Abstract

Machine learning applications such as finance and medicine demand accurate and justifiable predictions, barring most deep learning methods from use. In response, previous work combines decision trees with deep learning, yielding models that (1) sacrifice interpretability for accuracy or (2) sacrifice accuracy for interpretability. We forgo this dilemma by jointly improving accuracy and interpretability using Neural-Backed Decision Trees (NBDTs). NBDTs replace a neural network's final linear layer with a differentiable sequence of decisions and a surrogate loss. This forces the model to learn high-level concepts and lessens reliance on highly-uncertain decisions, yielding (1) accuracy: NBDTs match or outperform modern neural networks on CIFAR, ImageNet and better generalize to unseen classes by up to 16%. Furthermore, our surrogate loss improves the original model's accuracy by up to 2%. NBDTs also afford (2) interpretability: improving human trustby clearly identifying model mistakes and assisting in dataset debugging. Code and pretrained NBDTs are at https://github.com/alvinwan/neural-backed-decision-trees.

8 pages, 7 figures, accepted to ICLR 2021

References in corpus (14)

• Distilling the Knowledge in a Neural Network
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Towards A Rigorous Science of Interpretable Machine Learning
• Axiomatic Attribution for Deep Networks
• Wide Residual Networks
• Distilling a Neural Network Into a Soft Decision Tree
• CondConv: Conditionally Parameterized Convolutions for Efficient Inference
• RISE: Randomized Input Sampling for Explanation of Black-box Models
• Deciding How to Decide: Dynamic Routing in Artificial Neural Networks
• Effectiveness of Hierarchical Softmax in Large Scale Classification Tasks
• Network of Experts for Large-Scale Image Categorization
• SplineNets: Continuous Neural Decision Graphs
• Deep Convolutional Decision Jungle for Image Classification
• Learning Hierarchical Visual Representations in Deep Neural Networks Using Hierarchical Linguistic Labels

Cited by in corpus (11)

• Privacy-preserving Federated Adversarial Domain Adaption over Feature Groups for Interpretability
• CDT: Cascading Decision Trees for Explainable Reinforcement Learning
• VIPriors 1: Visual Inductive Priors for Data-Efficient Deep Learning Challenges
• SegNBDT: Visual Decision Rules for Segmentation
• Leveraging Sparse Linear Layers for Debuggable Deep Networks
• Coarse-to-Fine Curriculum Learning
• Dive into Decision Trees and Forests: A Theoretical Demonstration
• Harnessing value from data science in business: ensuring explainability and fairness of solutions
• Learning Representations for Axis-Aligned Decision Forests through Input Perturbation
• DRDF: Determining the Importance of Different Multimodal Information with Dual-Router Dynamic Framework
• Adaptive Verifiable Training Using Pairwise Class Similarity