4 papers · 1 filter
Profile-aware diagnostics of discarded parameter-Hessian mass
Maxim Bolshim, Alexander Kugaevskikh
Evaluating block-diagonal curvature models involves two questions: how much curvature they omit, and what a specific correction changes at a finite step. We measure the squared off…
Stochastic Estimation of the Layer-wise Hessian Trace for Monitoring Neural-network Training
Maxim Bolshim, Alexander Kugaevskikh
The loss and the norm of its gradient separate the healthy and the pathological regimes of neural-network training only weakly, whilst the curvature of the empirical risk differs q…
Inter-Layer Hessian Analysis of Neural Networks with DAG Architectures
Maxim Bolshim, Alexander Kugaevskikh
Automatic differentiation computes loss Hessians, while interpreting inter-layer interactions requires an explicit choice of graph-indexed blocks. This paper presents an analytical…
Local properties of neural networks through the lens of layer-wise Hessians
Maxim Bolshim, Alexander Kugaevskikh
We introduce a methodology for analyzing neural networks through the lens of layer-wise Hessian matrices. The local Hessian of each functional block (layer) is defined as the matri…