The Accuracy Well: Nonmonotone Accuracy Dynamics at Vanishing Initialization

Yaroslav Gusev, Eugene Golikov, Dmitry Yarotsky

We study classification dynamics under small initialization and identify a pronounced non-monotone behavior of accuracy that we call the accuracy well: accuracy rises rapidly early…