Block-diagonal Hessian-free Optimization for Training Neural Networks

Huishuai Zhang, Caiming Xiong, James Bradbury +1

Second-order methods for neural network optimization have several advantages over methods based on first-order gradient descent, including better scaling to large mini-batch sizes…