HARFE: Hard-Ridge Random Feature Expansion

Abstract

We propose a random feature model for approximating high-dimensional sparse additive functions called the hard-ridge random feature expansion method (HARFE). This method utilizes a hard-thresholding pursuit-based algorithm applied to the sparse ridge regression (SRR) problem to approximate the coefficients with respect to the random feature matrix. The SRR formulation balances between obtaining sparse models that use fewer terms in their representation and ridge-based smoothing that tend to be robust to noise and outliers. In addition, we use a random sparse connectivity pattern in the random feature matrix to match the additive function assumption. We prove that the HARFE method is guaranteed to converge with a given error bound depending on the noise and the parameters of the sparse ridge regression model. Based on numerical results on synthetic data as well as on real datasets, the HARFE approach obtains lower (or comparable) error than other state-of-the-art algorithms.

References in corpus (7)

• Reconciling modern machine learning practice and the bias-variance trade-off
• Component selection and smoothing in multivariate nonparametric regression
• Benign Overfitting in Linear Regression
• Just Interpolate: Kernel "Ridgeless" Regression Can Generalize
• Generalization Bounds for Sparse Random Feature Expansions
• Approximation of high-dimensional periodic functions with Fourier-based methods
• Interpretable Approximation of High-Dimensional Data

Cited by in corpus (1)

• Random Feature Models for Learning Interacting Dynamical Systems