Online Learning, Stability, and Stochastic Gradient Descent
arXiv:1105.4701
Abstract
In batch learning, stability together with existence and uniqueness of the solution corresponds to well-posedness of Empirical Risk Minimization (ERM) methods; recently, it was proved that CV_loo stability is necessary and sufficient for generalization and consistency of ERM. In this note, we introduce CV_on stability, which plays a similar note in online learning. We show that stochastic gradient descent (SDG) with the usual hypotheses is CVon stable and we then discuss the implications of CV_on stability for convergence of SGD.
11 pages
Cited by in corpus (8)
- Thompson sampling with the online bootstrap
- To Drop or Not to Drop: Robustness, Consistency and Differential Privacy Properties of Dropout
- Personalized Federated Learning for Heterogeneous Clients with Clustered Knowledge Transfer
- Generalization Error Bounds with Probabilistic Guarantee for SGD in Nonconvex Optimization
- The Interplay Between Stability and Regret in Online Learning
- Maximum likelihood estimation of a finite mixture of logistic regression models in a continuous data stream
- Lock in Feedback in Sequential Experiments
- Distributed SGD Generalizes Well Under Asynchrony