Generalization Properties and Implicit Regularization for Multiple Passes SGM
arXiv:1605.08375
Abstract
We study the generalization properties of stochastic gradient methods for learning with convex loss functions and linearly parameterized functions. We show that, in the absence of penalizations or constraints, the stability and approximation properties of the algorithm can be controlled by tuning either the step-size or the number of passes over the data. In this view, these parameters can be seen to control a form of implicit regularization. Numerical results complement the theoretical findings.
arXiv comment26 pages, 4 figures. To appear in ICML 2016
References in corpus (3)
Cited by in corpus (15)
- Implicit Regularization in Nonconvex Statistical Estimation: Gradient Descent Converges Linearly for Phase Retrieval, Matrix Completion, and Blind Deconvolution
- Implicit Regularization in Deep Matrix Factorization
- Training behavior of deep neural network in frequency domain
- The Implicit Regularization of Stochastic Gradient Flow for Least Squares
- Bad Global Minima Exist and SGD Can Reach Them
- Graph-Dependent Implicit Regularisation for Distributed Stochastic Subgradient Descent
- Stability and Generalization of Stochastic Gradient Methods for Minimax Problems
- Implicit Regularization of Accelerated Methods in Hilbert Spaces
- Can Implicit Bias Explain Generalization? Stochastic Convex Optimization as a Case Study
- Improved Learning Rates for Stochastic Optimization
- Learning with Gradient Descent and Weakly Convex Losses
- Stochastic Gradient Descent Meets Distribution Regression
- Adaptive Ensemble of Classifiers with Regularization for Imbalanced Data Classification
- A termination criterion for stochastic gradient descent for binary classification
- Why Does Multi-Epoch Training Help?