KrADagrad: Kronecker Approximation-Domination Gradient Preconditioned Stochastic Optimization

Jonathan Mei, Alexander Moreno, Luke Walters

Second order stochastic optimizers allow parameter update step size and direction to adapt to loss curvature, but have traditionally required too much memory and compute for deep l…