4 papers
Intrinsic training dynamics of deep neural networks
Sibylle Marcotte, Gabriel Peyré, Rémi Gribonval
A fundamental challenge in the theory of deep learning is to understand whether gradient-based training can promote parameters belonging to certain lower-dimensional structures (e.…
Transformative or Conservative? Conservation laws for ResNets and Transformers
Sibylle Marcotte, Rémi Gribonval, Gabriel Peyré
While conservation laws in gradient flow training dynamics are well understood for (mostly shallow) ReLU and linear networks, their study remains largely unexplored for more practi…
Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flows
Sibylle Marcotte, Rémi Gribonval, Gabriel Peyré
Understanding the geometric properties of gradient descent dynamics is a key ingredient in deciphering the recent success of very large machine learning models. A striking observat…
Keep the Momentum: Conservation Laws beyond Euclidean Gradient Flows
Sibylle Marcotte, Rémi Gribonval, Gabriel Peyré
Conservation laws are well-established in the context of Euclidean gradient flow dynamics, notably for linear or ReLU neural network training. Yet, their existence and principles f…