EnKSGD: A Class Of Preconditioned Black Box Optimization And Inversion Algorithms

Abstract

In this paper, we introduce the Ensemble Kalman-Stein Gradient Descent (EnKSGD) class of algorithms. The EnKSGD class of algorithms builds on the ensemble Kalman filter (EnKF) line of work, applying techniques from sequential data assimilation to unconstrained optimization and parameter estimation problems. The essential idea is to exploit the EnKF as a black box (i.e. derivative-free, zeroth order) optimization tool if iterated to convergence. In this paper, we return to the foundations of the EnKF as a sequential data assimilation technique, including its continuous-time and mean-field limits, with the goal of developing faster optimization algorithms suited to noisy black box optimization and inverse problems. The resulting EnKSGD class of algorithms can be designed to both maintain the desirable property of affine-invariance, and employ the well-known backtracking line search. Furthermore, EnKSGD algorithms are designed to not necessitate the subspace restriction property and variance collapse property of previous iterated EnKF approaches to optimization, as both these properties can be undesirable in an optimization context. EnKSGD also generalizes beyond the $L^2$ loss, and is thus applicable to a wider class of problems than the standard EnKF. Numerical experiments with both linear and nonlinear least squares problems, as well as maximum likelihood estimation, demonstrate the faster convergence of EnKSGD relative to alternative EnKF approaches to optimization.

arXiv comment20 pages, 3 figures

References in corpus (15)

• The Ensemble Kalman Filter for Inverse Problems
• Derivative-free optimization methods
• A consensus-based model for global optimization and its mean-field limit
• Ensemble Kalman Inversion: A Derivative-Free Technique For Machine Learning Tasks
• Well-Posedness And Accuracy Of The Ensemble Kalman Filter In Discrete And Continuous Time
• A regularizing iterative ensemble Kalman method for PDE-constrained inverse problems
• Parameterizations for Ensemble Kalman Inversion
• Nonlinear stability and ergodicity of ensemble based Kalman filters
• Convergence of the Square Root Ensemble Kalman Filter in the Large Ensemble Limit
• Ensemble Kalman Methods With Constraints
• Deterministic Mean-field Ensemble Kalman Filtering
• Well Posedness and Convergence Analysis of the Ensemble Kalman Inversion
• A mollified Ensemble Kalman filter
• Ensemble Kalman Inversion: mean-field limit and convergence analysis
• Ensemble Kalman Sampler: mean-field limit and convergence analysis