Thermodynamic Natural Gradient Descent

Kaelan Donatella, Samuel Duffield, Maxwell Aifer +3

Second-order training methods have better convergence properties than gradient descent but are rarely used in practice for large-scale training due to their computational overhead.…