Stochastic Gradient Descent in Hilbert Scales: Smoothness, Preconditioning and Earlier Stopping
arXiv:2006.10840
Abstract
Stochastic Gradient Descent (SGD) has become the method of choice for solving a broad range of machine learning problems. However, some of its learning properties are still not fully understood. We consider least squares learning in reproducing kernel Hilbert spaces (RKHSs) and extend the classical SGD analysis to a learning setting in Hilbert scales, including Sobolev spaces and Diffusion spaces on compact Riemannian manifolds. We show that even for well-specified models, violation of a traditional benchmark smoothness assumption has a tremendous effect on the learning rate. In addition, we show that for miss-specified models, preconditioning in an appropriate Hilbert scale helps to reduce the number of iterations, i.e. allowing for "earlier stopping".
References in corpus (9)
- Making Gradient Descent Optimal for Strongly Convex Stochastic Optimization
- Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)
- Better Mini-Batch Algorithms via Accelerated Gradient Methods
- Optimal Rates for Spectral Algorithms with Least-Squares Regression over Hilbert Spaces
- Optimal Rates for Multi-pass Stochastic Gradient Methods
- Statistical Optimality of Stochastic Gradient Descent on Hard Learning Problems through Multiple Passes
- Sobolev Norm Learning Rates for Regularized Least-Squares Algorithm
- Lepskii Principle in Supervised Learning
- Inverse learning in Hilbert scales