Component-Wise Natural Gradient Descent -- An Efficient Neural Network Optimization

Tran Van Sang, Mhd Irvan, Rie Shigetomi Yamaguchi +1

Natural Gradient Descent (NGD) is a second-order neural network training that preconditions the gradient descent with the inverse of the Fisher Information Matrix (FIM). Although N…