On the Weight Dynamics of Deep Normalized Networks

Christian H. X. Ali Mehmeti-Göpel, Michael Wand

Recent studies have shown that high disparities in effective learning rates (ELRs) across layers in deep neural networks can negatively affect trainability. We formalize how these…