Transformative or Conservative? Conservation laws for ResNets and Transformers

Sibylle Marcotte, Rémi Gribonval, Gabriel Peyré

While conservation laws in gradient flow training dynamics are well understood for (mostly shallow) ReLU and linear networks, their study remains largely unexplored for more practi…