TENGraD: Time-Efficient Natural Gradient Descent with Exact Fisher-Block Inversion

Saeed Soori, Bugra Can, Baourun Mu +2

This work proposes a time-efficient Natural Gradient Descent method, called TENGraD, with linear convergence guarantees. Computing the inverse of the neural network's Fisher inform…