Nonlinear Conjugate Gradients For Scaling Synchronous Distributed DNN Training

Saurabh Adya, Vinay Palakkode, Oncel Tuzel

Nonlinear conjugate gradient (NLCG) based optimizers have shown superior loss convergence properties compared to gradient descent based optimizers for traditional optimization prob…