Sparse/Robust Estimation and Kalman Smoothing with Nonsmooth Log-Concave Densities: Modeling, Computation, and Theory

Abstract

We introduce a class of quadratic support (QS) functions, many of which play a crucial role in a variety of applications, including machine learning, robust statistical inference, sparsity promotion, and Kalman smoothing. Well known examples include the l2, Huber, l1 and Vapnik losses. We build on a dual representation for QS functions using convex analysis, revealing the structure necessary for a QS function to be interpreted as the negative log of a probability density, and providing the foundation for statistical interpretation and analysis of QS loss functions. For a subclass of QS functions called piecewise linear quadratic (PLQ) penalties, we also develop efficient numerical estimation schemes. These components form a flexible statistical modeling framework for a variety of learning applications, together with a toolbox of efficient numerical methods for inference. In particular, for PLQ densities, interior point (IP) methods can be used. IP methods solve nonsmooth optimization problems by working directly with smooth systems of equations characterizing their optimality. The efficiency of the IP approach depends on the structure of particular applications. We consider the class of dynamic inverse problems using Kalman smoothing, where the aim is to reconstruct the state of a dynamical system with known process and measurement models starting from noisy output samples. In the classical case, Gaussian errors are assumed in the process and measurement models. The extended framework allows arbitrary PLQ densities to be used, and the proposed IP approach solves the generalized Kalman smoothing problem while maintaining the linear complexity in the size of the time series, just as in the Gaussian case. This extends the computational efficiency of classic algorithms to a much broader nonsmooth setting, and includes many recently proposed robust and sparse smoothers as special cases.

41 pages, 4 figures

References in corpus (1)

• Variational properties of value functions

Cited by in corpus (16)

• Dynamic Filtering of Time-Varying Sparse Signals via l1 Minimization
• Variational properties of value functions
• Maximum Correntropy Derivative-Free Robust Kalman Filter and Smoother
• Sparse Bayesian Learning with Dynamic Filtering for Inference of Time-Varying Sparse Signals
• 2-Wasserstein Approximation via Restricted Convex Potentials with Application to Improved Training for GANs
• Efficient evaluation of scaled proximal operators
• Non-smooth Variable Projection
• Sparse Quantile Huber Regression for Efficient and Robust Estimation
• Generalized Kalman Smoothing: Modeling and Algorithms
• Joint Maximum a Posteriori State Path and Parameter Estimation in Stochastic Differential Equations
• Visualization of the ε-Subdifferential of Piecewise Linear-Quadratic Functions
• Line Search and Trust-Region Methods for Convex-Composite Optimization
• l1-Norm Minimization with Regula Falsi Type Root Finding Methods
• Time Series Using Exponential Smoothing Cells
• Estimating Shape Parameters of Piecewise Linear-Quadratic Problems
• Log-Concave Duality in Estimation and Control