Stagewise Training Accelerates Convergence of Testing Error Over SGD

Zhuoning Yuan, Yan Yan, Rong Jin +1

Stagewise training strategy is widely used for learning neural networks, which runs a stochastic algorithm (e.g., SGD) starting with a relatively large step size (aka learning rate…