Binarsity: a penalization for one-hot encoded features in linear supervised learning

Abstract

This paper deals with the problem of large-scale linear supervised learning in settings where a large number of continuous features are available. We propose to combine the well-known trick of one-hot encoding of continuous features with a new penalization called \emph{binarsity}. In each group of binary features coming from the one-hot encoding of a single raw continuous feature, this penalization uses total-variation regularization together with an extra linear constraint. This induces two interesting properties on the model weights of the one-hot encoded features: they are piecewise constant, and are eventually block sparse. Non-asymptotic oracle inequalities for generalized linear models are proposed. Moreover, under a sparse additive model assumption, we prove that our procedure matches the state-of-the-art in this setting. Numerical experiments illustrate the good performances of our approach on several datasets. It is also noteworthy that our method has a numerical complexity comparable to standard ${\ell }_1$ penalization.

References in corpus (7)

• Scikit-learn: Machine Learning in Python
• Simultaneous analysis of Lasso and Dantzig selector
• Pathwise coordinate optimization
• Searching for Exotic Particles in High-Energy Physics with Deep Learning
• High-dimensional additive modeling
• On the Prediction Performance of the Lasso
• Tick: a Python library for statistical learning, with a particular emphasis on time-dependent modelling

Cited by in corpus (1)

• Does Configuration Encoding Matter in Learning Software Performance? An Empirical Study on Encoding Schemes