Cross-validated feature selection using least-squares support vector machines for binary classification

Abstract

Support vector machines (SVMs) are widely used for binary classification. However, irrelevant or redundant features can degrade classification accuracy and model interpretability, making feature selection important for SVMs. Most SVM feature selection methods rely on sequential search or regularization and do not directly identify the best feature subset. Mixed-integer optimization (MIO) is a direct approach to best subset selection. Cross-validation (CV) is broadly applicable to feature selection because it evaluates predictive performance on validation data without model assumptions. Although Purcaru et al. (2026) proposed an MIO formulation for CV-based linear-SVM feature selection (SVM-CV), this method suffers from high computational cost. We develop computationally efficient CV-based MIO formulations for feature selection in linear SVM classification by using least squares support vector machines (LS-SVMs). We formulate the problem as a bilevel MIO problem, where the lower-level problem trains LS-SVM classifiers and the upper-level problem selects features based on validation sets. We then reformulate it as a single-level MIO problem by exploiting the optimality conditions of LS-SVM. We propose two formulations: a hinge-loss-based validation model and a least-squares-based validation model. We compare our methods with SVM-CV and other SVM-based and logistic-regression-based methods through simulation experiments on synthetic data. The hinge-loss-based validation model attained high feature selection accuracy under the high-noise and moderate-noise conditions, particularly with more candidate features, and the least-squares-based validation model performed well in feature selection under the low-noise condition at the lower correlation level. Furthermore, our two models achieved substantially faster computation than the existing SVM-CV model.

arXiv comment25 pages, 5 figures, 1 table