Scalable and Practical Natural Gradient for Large-Scale Deep Learning

Kazuki Osawa, Yohei Tsuji, Yuichiro Ueno +3

Large-scale distributed training of deep neural networks results in models with worse generalization performance as a result of the increase in the effective mini-batch size. Previ…