Ensemble Kalman Inversion: A Derivative-Free Technique For Machine Learning Tasks

Abstract

The standard probabilistic perspective on machine learning gives rise to empirical risk-minimization tasks that are frequently solved by stochastic gradient descent (SGD) and variants thereof. We present a formulation of these tasks as classical inverse or filtering problems and, furthermore, we propose an efficient, gradient-free algorithm for finding a solution to these problems using ensemble Kalman inversion (EKI). Applications of our approach include offline and online supervised learning with deep neural networks, as well as graph-based semi-supervised learning. The essence of the EKI procedure is an ensemble based approximate gradient descent in which derivatives are replaced by differences from within the ensemble. We suggest several modifications to the basic method, derived from empirically successful heuristics developed in the context of SGD. Numerical results demonstrate wide applicability and robustness of the proposed algorithm.

arXiv comment41 pages, 14 figures

References in corpus (29)

• Adam: A Method for Stochastic Optimization
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• A Survey on Deep Learning in Medical Image Analysis
• Sequence to Sequence Learning with Neural Networks
• Explaining and Harnessing Adversarial Examples
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Intriguing properties of neural networks
• On the difficulty of training Recurrent Neural Networks
• Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
• Instance Normalization: The Missing Ingredient for Fast Stylization
• FitNets: Hints for Thin Deep Nets
• A Convergence Theory for Deep Learning via Over-Parameterization
• The Marginal Value of Adaptive Gradient Methods in Machine Learning
• Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning
• A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights
• Self-Normalizing Neural Networks
• The Ensemble Kalman Filter for Inverse Problems
• Fractional Max-Pooling
• Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)
• All you need is a good init
• Data Assimilation: A Mathematical Introduction
• Training Neural Networks Without Gradients: A Scalable ADMM Approach
• Gradient Descent Converges to Minimizers
• Entropy-SGD: Biasing Gradient Descent Into Wide Valleys
• Autoregressive Convolutional Neural Networks for Asynchronous Time Series
• Distributed optimization of deeply nested systems
• A mollified Ensemble Kalman filter
• Never look back - A modified EnKF method and its application to the training of neural networks without back propagation
• A Parallel Implementation of the Ensemble Kalman Filter Based on Modified Cholesky Decomposition

Cited by in corpus (29)

• Bridging observation, theory and numerical simulation of the ocean using Machine Learning
• Calibrate, Emulate, Sample
• Ensemble Kalman method for learning turbulence models from indirect observation data
• Well Posedness and Convergence Analysis of the Ensemble Kalman Inversion
• Towards physically consistent data-driven weather forecasting: Integrating data assimilation with equivariance-preserving deep spatial transformers
• Climbing down Charney's ladder: Machine Learning and the post-Dennard era of computational climate science
• Online learning of both state and dynamics using ensemble Kalman filters
• Analysis of Stochastic Gradient Descent in Continuous Time
• Explainable Offline-Online Training of Neural Networks for Parameterizations: A 1D Gravity Wave-QBO Testbed in the Small-data Regime
• Localization in Ensemble Kalman inversion
• Adaptive Tikhonov strategies for stochastic ensemble Kalman inversion
• Neural Dynamical Operator: Continuous Spatial-Temporal Model with Gradient-Based and Derivative-Free Optimization Methods
• Non-Asymptotic Analysis of Ensemble Kalman Updates: Effective Dimension and Localization
• On convergence rates of adaptive ensemble Kalman inversion for linear ill-posed problems
• Hyperparameter Optimization for Randomized Algorithms: A Case Study on Random Features
• Gradient flow structure and convergence analysis of the ensemble Kalman inversion for nonlinear forward models
• Subsampling in ensemble Kalman inversion
• Improve Unscented Kalman Inversion With Low-Rank Approximation and Reduced-Order Model
• Dropout Ensemble Kalman inversion for high dimensional inverse problems
• Nesterov Acceleration for Ensemble Kalman Inversion and Variants
• Active Learning of Model Discrepancy with Bayesian Experimental Design
• Covariance-modulated optimal transport and gradient flows
• The sharp, the flat and the shallow: Can weakly interacting agents learn to escape bad minima?
• Solving continuum and rarefied flows using differentiable programming
• Sequential Kalman Tuning of the $t$-preconditioned Crank-Nicolson algorithm: efficient, adaptive and gradient-free inference for Bayesian inverse problems
• Sampling error correction in ensemble Kalman inversion
• Adaptive regularisation for ensemble Kalman inversion
• EnKSGD: A Class Of Preconditioned Black Box Optimization And Inversion Algorithms
• Ensemble gradient for learning turbulence models from indirect observations