An Ensemble Information Filter: Learning and Encoding Sparse Structures in Ensemble Kalman Methods

Abstract

Ensemble-based Data Assimilation faces significant challenges in high-dimensional systems due to spurious correlations and ensemble collapse. We propose the Ensemble Information Filter, which addresses these issues by encoding locality directly in the precision matrix through sparse Markov structure. This approach is motivated by connections to SPDE-based models, where local operators induce sparse precision representations and provide a principled form of regularisation. The framework further incorporates sparse regression for learning the observation operator, enabling stable inference in regimes where $p\gg n$. By enforcing structure prior to estimation, EnIF avoids the need for heuristic localisation and improves both statistical and computational scalability. The methodology is demonstrated on filtering, smoothing, and parameter estimation problems, where it adapts to the underlying dependence structure and yields stable and efficient inference in high-dimensional systems.

28 pages, 10 figures