Stabilize Deep ResNet with A Sharp Scaling Factor $\tau$

Huishuai Zhang, Da Yu, Mingyang Yi +2

We study the stability and convergence of training deep ResNets with gradient descent. Specifically, we show that the parametric branch in the residual block should be scaled down…