Accelerating SGD for Distributed Deep-Learning Using Approximated Hessian Matrix

Sébastien M. R. Arnold, Chunming Wang

We introduce a novel method to compute a rank $m$ approximation of the inverse of the Hessian matrix in the distributed regime. By leveraging the differences in gradients and param…