Decision Machines: Congruent Decision Trees
arXiv:2101.11347
Abstract
The decision tree recursively partitions the input space into regions and derives axis-aligned decision boundaries from data. Despite its simplicity and interpretability, decision trees lack parameterized representation, which makes it prone to overfitting and difficult to find the optimal structure. We propose Decision Machines, which embed Boolean tests into a binary vector space and represent the tree structure as a matrices, enabling an interleaved traversal of decision trees through matrix computation. Furthermore, we explore the congruence of decision trees and attention mechanisms, opening new avenues for optimizing decision trees and potentially enhancing their predictive power.
References in corpus (13)
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Categorical Reparameterization with Gumbel-Softmax
- From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification
- Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
- Bayesian Hierarchical Mixtures of Experts
- Deep Neural Decision Trees
- A Regularized Framework for Sparse and Structured Neural Attention
- Efficient non-greedy optimization of decision trees
- Learning Piece-wise Linear Models from Large Scale Data for Ad Click Prediction
- Generalised Linear Model Trees with Global Additive Effects
- On Controllable Sparse Alternatives to Softmax
- Sigsoftmax: Reanalysis of the Softmax Bottleneck
- Oblique Decision Trees from Derivatives of ReLU Networks