Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networks

Abstract

In nature, symmetry governs regularities, while symmetry breaking brings texture. In artificial neural networks, symmetry has been a central design principle to efficiently capture regularities in the world, but the role of symmetry breaking is not well understood. Here, we develop a theoretical framework to study the "geometry of learning dynamics" in neural networks, and reveal a key mechanism of explicit symmetry breaking behind the efficiency and stability of modern neural networks. To build this understanding, we model the discrete learning dynamics of gradient descent using a continuous-time Lagrangian formulation, in which the learning rule corresponds to the kinetic energy and the loss function corresponds to the potential energy. Then, we identify "kinetic symmetry breaking" (KSB), the condition when the kinetic energy explicitly breaks the symmetry of the potential function. We generalize Noether's theorem known in physics to take into account KSB and derive the resulting motion of the Noether charge: "Noether's Learning Dynamics" (NLD). Finally, we apply NLD to neural networks with normalization layers and reveal how KSB introduces a mechanism of "implicit adaptive optimization", establishing an analogy between learning dynamics induced by normalization layers and RMSProp. Overall, through the lens of Lagrangian mechanics, we have established a theoretical foundation to discover geometric design principles for the learning dynamics of neural networks.

References in corpus (29)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges
• A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights
• A Variational Perspective on Accelerated Methods in Optimization
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• New insights and perspectives on the natural gradient method
• L2 Regularization versus Batch and Weight Normalization
• On Empirical Comparisons of Optimizers for Deep Learning
• Path-SGD: Path-Normalized Optimization in Deep Neural Networks
• A Lyapunov Analysis of Momentum Methods in Optimization
• On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization
• Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks
• Norm matters: efficient and accurate normalization schemes in deep networks
• Theoretical Analysis of Auto Rate-Tuning by Batch Normalization
• The large learning rate phase of deep learning: the catapult mechanism
• Limitations of the Empirical Fisher Approximation for Natural Gradient Descent
• Understanding self-supervised Learning Dynamics without Contrastive Pairs
• Scalable Second Order Optimization for Deep Learning
• Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability
• Understanding symmetries in deep networks
• On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)
• How to decay your learning rate
• Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances
• Continuous Time Analysis of Momentum Methods
• The Two Regimes of Deep Network Training
• A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizes
• A Coordinate-Free Construction of Scalable Natural Gradient
• Stochastic Training is Not Necessary for Generalization
• Accelerating Natural Gradient with Higher-Order Invariance