3 papers
cs.LG2022
Brand New K-FACs: Speeding up K-FAC with Online Decomposition Updates
Constantin Octavian Puiu
K-FAC (arXiv:1503.05671, arXiv:1602.01407) is a tractable implementation of Natural Gradient (NG) for Deep Learning (DL), whose bottleneck is computing the inverses of the so-calle…
cs.LG2022
Randomized K-FACs: Speeding up K-FAC with Randomized Numerical Linear Algebra
Constantin Octavian Puiu
K-FAC is a successful tractable implementation of Natural Gradient for Deep Learning, which nevertheless suffers from the requirement to compute the inverse of the Kronecker factor…
cs.LG2022
Rethinking Exponential Averaging of the Fisher
Constantin Octavian Puiu
In optimization for Machine learning (ML), it is typical that curvature-matrix (CM) estimates rely on an exponential average (EA) of local estimates (giving EA-CM algorithms). This…