Optimization Methods for Interpretable Differentiable Decision Trees in Reinforcement Learning

Abstract

Decision trees are ubiquitous in machine learning for their ease of use and interpretability. Yet, these models are not typically employed in reinforcement learning as they cannot be updated online via stochastic gradient descent. We overcome this limitation by allowing for a gradient update over the entire tree that improves sample complexity affords interpretable policy extraction. First, we include theoretical motivation on the need for policy-gradient learning by examining the properties of gradient descent over differentiable decision trees. Second, we demonstrate that our approach equals or outperforms a neural network on all domains and can learn discrete decision trees online with average rewards up to 7x higher than a batch-trained decision tree. Third, we conduct a user study to quantify the interpretability of a decision tree, rule list, and a neural network with statistically significant results ($p<0.001$).

References in corpus (11)

• Scikit-learn: Machine Learning in Python
• Playing Atari with Deep Reinforcement Learning
• Towards A Rigorous Science of Interpretable Machine Learning
• Interpretable classifiers using rules and Bayesian analysis: Building a better stroke prediction model
• StarCraft II: A New Challenge for Reinforcement Learning
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Learning Dexterous In-Hand Manipulation
• TStarBots: Defeating the Cheating Level Builtin AI in StarCraft II in the Full Game
• The Thing That We Tried Didn't Work Very Well : Deictic Representation in Reinforcement Learning
• An Optimization Approach to Learning Falling Rule Lists
• Neural-encoding Human Experts' Domain Knowledge to Warm Start Reinforcement Learning

Cited by in corpus (6)

• CDT: Cascading Decision Trees for Explainable Reinforcement Learning
• Feature-Based Interpretable Reinforcement Learning based on State-Transition Models
• Deep differentiable forest with sparse attention for the tabular data
• Deep Inductive Logic Programming meets Reinforcement Learning
• Attention augmented differentiable forest for tabular data
• A short note on the decision tree based neural turing machine