Factorization Machines with Regularization for Sparse Feature Interactions

Abstract

Factorization machines (FMs) are machine learning predictive models based on second-order feature interactions and FMs with sparse regularization are called sparse FMs. Such regularizations enable feature selection, which selects the most relevant features for accurate prediction, and therefore they can contribute to the improvement of the model accuracy and interpretability. However, because FMs use second-order feature interactions, the selection of features often causes the loss of many relevant feature interactions in the resultant models. In such cases, FMs with regularization specially designed for feature interaction selection trying to achieve interaction-level sparsity may be preferred instead of those just for feature selection trying to achieve feature-level sparsity. In this paper, we present a new regularization scheme for feature interaction selection in FMs. The proposed regularizer is an upper bound of the ${\ell }_1$ regularizer for the feature interaction matrix, which is computed from the parameter matrix of FMs. For feature interaction selection, our proposed regularizer makes the feature interaction matrix sparse without a restriction on sparsity patterns imposed by the existing methods. We also describe efficient proximal algorithms for the proposed FMs and present theoretical analyses of both existing and the new regularize. In addition, we will discuss how our ideas can be applied or extended to more accurate feature selection and other related models such as higher-order FMs and the all-subsets model. The analysis and experimental results on synthetic and real-world datasets show the effectiveness of the proposed methods.

References in corpus (9)

• A note on the group lasso and a sparse group lasso
• Convergence Rates of Inexact Proximal-Gradient Methods for Convex Optimization
• Estimation of Simultaneously Sparse and Low Rank Matrices
• Higher-Order Factorization Machines
• Polynomial Networks and Factorization Machines: New Insights and Efficient Training Algorithms
• Dual Iterative Hard Thresholding: From Non-convex Sparse Minimization to Non-smooth Concave Maximization
• The Kernel Interaction Trick: Fast Bayesian Discovery of Pairwise Interactions in High Dimensions
• RaFM: Rank-Aware Factorization Machines
• Interaction Hard Thresholding: Consistent Sparse Quadratic Regression in Sub-quadratic Time and Space