Is All Learning (Natural) Gradient Descent?

Lucas Shoji, Kenta Suzuki, Leo Kozachkov

This paper shows that a wide class of effective learning rules -- those that improve a scalar performance measure over a given time window -- can be rewritten as natural gradient d…