Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics

Abstract

Understanding the dynamics of neural network parameters during training is one of the key challenges in building a theoretical foundation for deep learning. A central obstacle is that the motion of a network in high-dimensional parameter space undergoes discrete finite steps along complex stochastic gradients derived from real-world datasets. We circumvent this obstacle through a unifying theoretical framework based on intrinsic symmetries embedded in a network's architecture that are present for any dataset. We show that any such symmetry imposes stringent geometric constraints on gradients and Hessians, leading to an associated conservation law in the continuous-time limit of stochastic gradient descent (SGD), akin to Noether's theorem in physics. We further show that finite learning rates used in practice can actually break these symmetry induced conservation laws. We apply tools from finite difference methods to derive modified gradient flow, a differential equation that better approximates the numerical trajectory taken by SGD at finite learning rates. We combine modified gradient flow with our framework of symmetries to derive exact integral expressions for the dynamics of certain parameter combinations. We empirically validate our analytic expressions for learning dynamics on VGG-16 trained on Tiny ImageNet. Overall, by exploiting symmetry, our work demonstrates that we can analytically describe the learning dynamics of various parameter combinations at finite learning rates and batch sizes for state of the art architectures trained on any dataset.

arXiv comment30 pages, 17 figures, ICLR 2021

References in corpus (17)

• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• A Differential Equation for Modeling Nesterov's Accelerated Gradient Method: Theory and Insights
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• A mathematical theory of semantic development in deep neural networks
• Three Factors Influencing Minima in SGD
• L2 Regularization versus Batch and Weight Normalization
• Path-SGD: Path-Normalized Optimization in Deep Neural Networks
• On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization
• A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks
• Norm matters: efficient and accurate normalization schemes in deep networks
• The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects
• Three Mechanisms of Weight Decay Regularization
• An Exponential Learning Rate Schedule for Deep Learning
• Width Provably Matters in Optimization for Deep Linear Neural Networks
• On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)
• Uniform-in-Time Weak Error Analysis for Stochastic Gradient Descent Algorithms via Diffusion Approximation

Cited by in corpus (5)

• Automatic Symmetry Discovery with Lie Algebra Convolutional Network
• On the Validity of Modeling SGD with Stochastic Differential Equations (SDEs)
• Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances
• SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticality
• The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusion