Regularization Learning Networks: Deep Learning for Tabular Datasets

Abstract

Despite their impressive performance, Deep Neural Networks (DNNs) typically underperform Gradient Boosting Trees (GBTs) on many tabular-dataset learning tasks. We propose that applying a different regularization coefficient to each weight might boost the performance of DNNs by allowing them to make more use of the more relevant inputs. However, this will lead to an intractable number of hyperparameters. Here, we introduce Regularization Learning Networks (RLNs), which overcome this challenge by introducing an efficient hyperparameter tuning scheme which minimizes a new Counterfactual Loss. Our results show that RLNs significantly improve DNNs on tabular datasets, and achieve comparable results to GBTs, with the best performance achieved with an ensemble that combines GBTs and RLNs. RLNs produce extremely sparse networks, eliminating up to 99.8% of the network edges and 82% of the input features, thus providing more interpretable models and reveal the importance that the network assigns to different inputs. RLNs could efficiently learn a single network in datasets that comprise both tabular and unstructured data, such as in the setting of medical imaging accompanied by electronic health records. An open source implementation of RLN can be found at https://github.com/irashavitt/regularization_learning_networks.

arXiv commentAccepted to the 32nd Conference on Neural Information Processing Systems (NIPS 2018), Montreal, Canada

References in corpus (9)

• Explaining and Harnessing Adversarial Examples
• Practical Bayesian Optimization of Machine Learning Algorithms
• European Union regulations on algorithmic decision-making and a "right to explanation"
• A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay
• Not Just a Black Box: Learning Important Features Through Propagating Activation Differences
• Gradient-based Hyperparameter Optimization through Reversible Learning
• Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation
• Representation Learning: A Review and New Perspectives
• Stochastic Hyperparameter Optimization through Hypernetworks

Cited by in corpus (2)

• SAINT: Improved Neural Networks for Tabular Data via Row Attention and Contrastive Pre-Training
• A Comparative Study of Machine Learning Models for Tabular Data Through Challenge of Monitoring Parkinson's Disease Progression Using Voice Recordings