Kernelized Wasserstein Natural Gradient
arXiv:1910.09652
Abstract
Many machine learning problems can be expressed as the optimization of some cost functional over a parametric family of probability distributions. It is often beneficial to solve such optimization problems using natural gradient methods. These methods are invariant to the parametrization of the family, and thus can yield more effective optimization. Unfortunately, computing the natural gradient is challenging as it requires inverting a high dimensional matrix at each iteration. We propose a general framework to approximate the natural gradient for the Wasserstein metric, by leveraging a dual formulation of the metric restricted to a Reproducing Kernel Hilbert Space. Our approach leads to an estimator for gradient direction that can trade-off accuracy and computational cost, with theoretical guarantees. We verify its accuracy on simple examples, and show the advantage of using such an estimator in classification tasks on Cifar10 and Cifar100 empirically.
References in corpus (11)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Optimizing Neural Networks with Kronecker-factored Approximate Curvature
- Information-Geometric Optimization Algorithms: A Unifying Picture via Invariance Principles
- Density Estimation in Infinite Dimensional Exponential Families
- Fast Approximate Natural Gradient Descent in a Kronecker-factored Eigenbasis
- Large sample analysis of the median heuristic
- SLANG: Fast Structured Covariance Approximations for Bayesian Deep Learning with Natural Gradient
- Ricci curvature for parametric statistics via optimal transport
- Wasserstein Riemannian Geometry of Positive Definite Matrices
- The nonparametric Fisher geometry and the chi-square process density prior