Analysis and Comparison of Two-Level KFAC Methods for Training Deep Neural Networks

Abdoulaye Koroko, Ani Anciaux-Sedrakian, Ibtihel Ben Gharbia +3

As a second-order method, the Natural Gradient Descent (NGD) has the ability to accelerate training of neural networks. However, due to the prohibitive computational and memory cos…