Noether: The More Things Change, the More Stay the Same
arXiv:2104.05508
Abstract
Symmetries have proven to be important ingredients in the analysis of neural networks. So far their use has mostly been implicit or seemingly coincidental. We undertake a systematic study of the role that symmetry plays. In particular, we clarify how symmetry interacts with the learning algorithm. The key ingredient in our study is played by Noether's celebrated theorem which, informally speaking, states that symmetry leads to conserved quantities (e.g., conservation of energy or conservation of momentum). In the realm of neural networks under gradient descent, model symmetries imply restrictions on the gradient path. E.g., we show that symmetry of activation functions leads to boundedness of weight matrices, for the specific case of linear activations it leads to balance equations of consecutive layers, data augmentation leads to gradient paths that have "momentum"-type restrictions, and time symmetry leads to a version of the Neural Tangent Kernel. Symmetry alone does not specify the optimization path, but the more symmetries are contained in the model the more restrictions are imposed on the path. Since symmetry also implies over-parametrization, this in effect implies that some part of this over-parametrization is cancelled out by the existence of the conserved quantities. Symmetry can therefore be thought of as one further important tool in understanding the performance of neural networks under gradient descent.
arXiv comment35 pages, 6 figures
References in corpus (20)
- Reconciling modern machine learning practice and the bias-variance trade-off
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights
- Gradient Descent Provably Optimizes Over-parameterized Neural Networks
- Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
- Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
- Spectrally-normalized margin bounds for neural networks
- The loss surface of deep and wide neural networks
- On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization
- A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks
- Global Optimality in Tensor Factorization, Deep Learning, and Beyond
- When Do Neural Networks Outperform Kernel Methods?
- Identity Matters in Deep Learning
- Implicit Regularization in Deep Matrix Factorization
- Global optimality conditions for deep neural networks
- On the Power and Limitations of Random Features for Understanding Neural Networks
- Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balanced
- Optimization and Generalization of Shallow Neural Networks with Quadratic Activation Functions
- Learning Parities with Neural Networks
- Emmy Noether on Energy Conservation in General Relativity