The Hessian of tall-skinny networks is easy to invert

Ali Rahimi

We describe an exact algorithm to solve linear systems of the form $Hx=b$ where $H$ is the Hessian of a deep net. The method computes Hessian-inverse-vector products without storin…