Depth Dependence of $\mu$P Learning Rates in ReLU MLPs

Samy Jelassi, Boris Hanin, Ziwei Ji +3

In this short note we consider random fully connected ReLU networks of width $n$ and depth $L$ equipped with a mean-field weight initialization. Our purpose is to study the depende…