When Both Layers Learn: Training Dynamics of Representing Linear Models via ReLU Networks

Berk Tinaz, Changzhi Xie, Mahdi Soltanolkotabi

In this paper, we study the gradient descent dynamics for jointly training both layers of a one-hidden-layer ReLU network to fit a linear target function. Concretely, we consider a…