L1 Regularization Paths in Linear Models by Parametric Gaussian Message Passing

Abstract

The paper proposes new algorithms for computing the regularization path of linear-model fitting (i) with L1 costs for the independent variables and L2 costs for the dependent variables, and (ii) with L2 costs for the independent variables and L1 costs for the dependent variables. Unlike previously proposed algorithms, the proposed algorithms for (i) and (ii) are exact duals of each other. The heart of the proposed algorithms is parametric Gaussian message passing (i.e., Kalman-type forward-backward recursions) in the pertinent factor graphs. We first develop such algorithms for linear models with general (but preferably low-dimensional) state space representations. We then specialize to specific state space representations of generic linear models; in this case, the structure of the factor graph allows to refine the proposed algorithms such that the computational complexity drops to linear scale.