Magnitude and Angle Dynamics in Training Single ReLU Neurons

Sangmin Lee, Byeongsu Sim, Jong Chul Ye

To understand learning the dynamics of deep ReLU networks, we investigate the dynamic system of gradient flow $w(t)$ by decomposing it to magnitude $w(t)$ and angle $ϕ(t):= π- θ(t)…