Small steps and giant leaps: Minimal Newton solvers for Deep Learning

João F. Henriques, Sebastien Ehrhardt, Samuel Albanie +1

We propose a fast second-order method that can be used as a drop-in replacement for current deep learning solvers. Compared to stochastic gradient descent (SGD), it only requires t…