Non asymptotic analysis of Adaptive stochastic gradient algorithms and applications
arXiv:2303.01370
Abstract
In stochastic optimization, a common tool to deal sequentially with large sample is to consider the well-known stochastic gradient algorithm. Nevertheless, since the stepsequence is the same for each direction, this can lead to bad results in practice in case of ill-conditionned problem. To overcome this, adaptive gradient algorithms such that Adagrad or Stochastic Newton algorithms should be prefered. This paper is devoted to the non asymptotic analyis of these adaptive gradient algorithms for strongly convex objective. All the theoretical results will be adapted to linear regression and regularized generalized linear model for both Adagrad and Stochastic Newton algorithms.
References in corpus (9)
- Mini-Batch Semi-Stochastic Gradient Descent in the Proximal Setting
- Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)
- Efficient and fast estimation of the geometric median in Hilbert spaces with an averaged stochastic gradient algorithm
- Adaptivity of averaged stochastic gradient descent to local strong convexity for logistic regression
- Optimal non-asymptotic bound of the Ruppert-Polyak averaging without strong convexity
- An efficient Averaged Stochastic Gauss-Newton algorithm for estimating parameters of non linear regressions models
- Stochastic Online Optimization using Kalman Recursion
- Estimating the geometric median in Hilbert spaces with stochastic gradient algorithms: and almost sure rates of convergence
- Convergence in quadratic mean of averaged stochastic gradient algorithms without strong convexity nor bounded gradient