Depth Without the Magic: Inductive Bias of Natural Gradient Descent

Abstract

In gradient descent, changing how we parametrize the model can lead to drastically different optimization trajectories, giving rise to a surprising range of meaningful inductive biases: identifying sparse classifiers or reconstructing low-rank matrices without explicit regularization. This implicit regularization has been hypothesised to be a contributing factor to good generalization in deep learning. However, natural gradient descent is approximately invariant to reparameterization, it always follows the same trajectory and finds the same optimum. The question naturally arises: What happens if we eliminate the role of parameterization, which solution will be found, what new properties occur? We characterize the behaviour of natural gradient flow in deep linear networks for separable classification under logistic loss and deep matrix factorization. Some of our findings extend to nonlinear neural networks with sufficient but finite over-parametrization. We demonstrate that there exist learning problems where natural gradient descent fails to generalize, while gradient descent with the right architecture performs well.

References in corpus (14)

• Deep Residual Learning for Image Recognition
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• New insights and perspectives on the natural gradient method
• Deep learning generalizes because the parameter-function map is biased towards simple functions
• Kernel and Rich Regimes in Overparametrized Models
• Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networks
• Implicit Regularization for Optimal Sparse Recovery
• When Does Preconditioning Help or Hurt Generalization?
• A Coordinate-Free Construction of Scalable Natural Gradient
• Accelerating Natural Gradient with Higher-Order Invariance
• Stochastic Training is Not Necessary for Generalization
• Kernelized Wasserstein Natural Gradient
• On the proliferation of support vectors in high dimensions