Convergence Properties of Natural Gradient Descent for Minimizing KL Divergence

Adwait Datar, Nihat Ay

The Kullback-Leibler (KL) divergence plays a central role in probabilistic machine learning, where it commonly serves as the canonical loss function. Optimization in such settings…