Unnatural Algorithms in Machine Learning

Christian Goodbrake

Natural gradient descent has a remarkable property that in the small learning rate limit, it displays an invariance with respect to network reparameterizations, leading to robust t…