A Coordinate-Free Construction of Scalable Natural Gradient

Abstract

Most neural networks are trained using first-order optimization methods, which are sensitive to the parameterization of the model. Natural gradient descent is invariant to smooth reparameterizations because it is defined in a coordinate-free way, but tractable approximations are typically defined in terms of coordinate systems, and hence may lose the invariance properties. We analyze the invariance properties of the Kronecker-Factored Approximate Curvature (K-FAC) algorithm by constructing the algorithm in a coordinate-free way. We explicitly construct a Riemannian metric under which the natural gradient matches the K-FAC update; invariance to affine transformations of the activations follows immediately. We extend our framework to analyze the invariance properties of K-FAC applied to convolutional networks and recurrent neural networks, as well as metrics other than the usual Fisher metric.

References in corpus (10)

• Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• New insights and perspectives on the natural gradient method
• Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting
• Krylov Subspace Descent for Deep Learning
• Learning Feature Hierarchies with Centered Deep Boltzmann Machines
• Natural Neural Networks
• A Kronecker-factored approximate Fisher matrix for convolution layers
• Noisy Natural Gradient as Variational Inference
• Accelerating Natural Gradient with Higher-Order Invariance

Cited by in corpus (6)

• Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networks
• Three Mechanisms of Weight Decay Regularization
• An Empirical Study of Large-Batch Stochastic Gradient Descent with Structured Covariance Noise
• Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networks
• Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shaping
• Depth Without the Magic: Inductive Bias of Natural Gradient Descent