46 citations · 153 across the 11 of their papers we have counts for
3 papers · 1 filter
AsymptoticNG: A regularized natural gradient optimization algorithm with look-ahead strategy
Zedong Tang, Fenlong Jiang, Junke Song +5
Optimizers that further adjust the scale of gradient, such as Adam, Natural Gradient (NG), etc., despite widely concerned and used by the community, are often found poor generaliza…
Eigenvalue-corrected Natural Gradient Based on a New Approximation
Kai-Xin Gao, Xiao-Lei Liu, Zheng-Hai Huang +5
Using second-order optimization methods for training deep neural networks (DNNs) has attracted many researchers. A recently proposed method, Eigenvalue-corrected Kronecker Factoriz…
A Trace-restricted Kronecker-Factored Approximation to Natural Gradient
Kai-Xin Gao, Xiao-Lei Liu, Zheng-Hai Huang +4
Second-order optimization methods have the ability to accelerate convergence by modifying the gradient through the curvature matrix. There have been many attempts to use second-ord…