Feature Selection using Stochastic Gates

Abstract

Feature selection problems have been extensively studied for linear estimation, for instance, Lasso, but less emphasis has been placed on feature selection for non-linear functions. In this study, we propose a method for feature selection in high-dimensional non-linear function estimation problems. The new procedure is based on minimizing the ${\ell }_0$ norm of the vector of indicator variables that represent if a feature is selected or not. Our approach relies on the continuous relaxation of Bernoulli distributions, which allows our model to learn the parameters of the approximate Bernoulli distributions via gradient descent. This general framework simultaneously minimizes a loss function while selecting relevant features. Furthermore, we provide an information-theoretic justification of incorporating Bernoulli distribution into our approach and demonstrate the potential of the approach on synthetic and real-life applications.

arXiv commentPublished in ICML 2020

Cited by in corpus (8)

• Differentiable Unsupervised Feature Selection based on a Gated Laplacian
• Learning Deep Attribution Priors Based On Prior Knowledge
• Towards Rigorous Interpretations: a Formalisation of Feature Attribution
• Few-shot Learning for Unsupervised Feature Selection
• ${\ell }_0$-based Sparse Canonical Correlation Analysis
• Top-$k$ Regularization for Supervised Feature Selection
• Beyond Discriminant Patterns: On the Robustness of Decision Rule Ensembles
• Support Recovery with Stochastic Gates: Theory and Application for Linear Models