AsymptoticNG: A regularized natural gradient optimization algorithm with look-ahead strategy

Zedong Tang, Fenlong Jiang, Junke Song +5

Optimizers that further adjust the scale of gradient, such as Adam, Natural Gradient (NG), etc., despite widely concerned and used by the community, are often found poor generaliza…