Practical Knowledge Distillation: Using DNNs to Beat DNNs

Chung-Wei Lee, Pavlos Athanasios Apostolopulos, Igor L. Markov

For tabular data sets, we explore data and model distillation, as well as data denoising. These techniques improve both gradient-boosting models and a specialized DNN architecture.…