Muddling Label Regularization: Deep Learning for Tabular Datasets
arXiv:2106.04462
Abstract
Deep Learning (DL) is considered the state-of-the-art in computer vision, speech recognition and natural language processing. Until recently, it was also widely accepted that DL is irrelevant for learning tasks on tabular data, especially in the small sample regime where ensemble methods are acknowledged as the gold standard. We present a new end-to-end differentiable method to train a standard FFNN. Our method, \textbf{Muddling labels for Regularization} (\texttt{MLR}), penalizes memorization through the generation of uninformative labels and the application of a differentiable close-form regularization scheme on the last hidden layer during training. \texttt{MLR} outperforms classical NN and the gold standard (GBDT, RF) for regression and classification tasks on several datasets from the UCI database and Kaggle covering a large range of sample sizes and feature to sample ratios. Researchers and practitioners can use \texttt{MLR} on its own as an off-the-shelf \DL{} solution or integrate it into the most advanced ML pipelines.
arXiv comment36 pages
References in corpus (12)
- Understanding deep learning requires rethinking generalization
- fastai: A Layered API for Deep Learning
- A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay
- Entity Embeddings of Categorical Variables
- Narrowing the Gap: Random Forests In Theory and In Practice
- Deep Neural Decision Trees
- Applying Deep Learning To Airbnb Search
- Multi-Layered Gradient Boosting Decision Trees
- Deep Learning over Multi-field Categorical Data: A Case Study on User Response Prediction
- Deep Interest Network for Click-Through Rate Prediction
- Forward Thinking: Building Deep Random Forests
- Muddling Labels for Regularization, a novel approach to generalization