Why Does Multi-Epoch Training Help?

Abstract

Stochastic gradient descent (SGD) has become the most attractive optimization method in training large-scale deep neural networks due to its simplicity, low computational cost in each updating step, and good performance. Standard excess risk bounds show that SGD only needs to take one pass over the training data and more passes could not help to improve the performance. Empirically, it has been observed that SGD taking more than one pass over the training data (multi-pass SGD) has much better excess risk bound performance than the SGD only taking one pass over the training data (one-pass SGD). However, it is not very clear that how to explain this phenomenon in theory. In this paper, we provide some theoretical evidences for explaining why multiple passes over the training data can help improve performance under certain circumstance. Specifically, we consider smooth risk minimization problems whose objective function is non-convex least squared loss. Under Polyak-Lojasiewicz (PL) condition, we establish faster convergence rate of excess risk bound for multi-pass SGD than that for one-pass SGD.

References in corpus (10)

• A Convergence Theory for Deep Learning via Over-Parameterization
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks
• On Lazy Training in Differentiable Programming
• A Simple Proximal Stochastic Gradient Method for Nonsmooth Nonconvex Optimization
• Data-Dependent Stability of Stochastic Gradient Descent
• Small ReLU networks are powerful memorizers: a tight analysis of memorization capacity
• Optimal Rates for Multi-pass Stochastic Gradient Methods
• Generalization Properties and Implicit Regularization for Multiple Passes SGM
• Generalization Error Bounds with Probabilistic Guarantee for SGD in Nonconvex Optimization