Scalable K-FAC Training for Deep Neural Networks with Distributed Preconditioning

Lin Zhang, Shaohuai Shi, Wei Wang +1

The second-order optimization methods, notably the D-KFAC (Distributed Kronecker Factored Approximate Curvature) algorithms, have gained traction on accelerating deep neural networ…