Towards Practical Second-Order Optimizers in Deep Learning: Insights from Fisher Information Analysis

Damien Martins Gomes

First-order optimization methods remain the standard for training deep neural networks (DNNs). Optimizers like Adam incorporate limited curvature information by preconditioning the…