5 papers
Stochastic Gradient Descent: A Nonlinear Time Series Perspective
Jiaqi Li, Zhipeng Lou, Stefan Richter +1
This paper investigates the statistical behavior of Stochastic Gradient Descent (SGD) with constant step size under the framework of iterated random functions. Compared to previous…
Statistical Guarantees for High-Dimensional Stochastic Gradient Descent
Jiaqi Li, Zhipeng Lou, Johannes Schmidt-Hieber +1
Stochastic Gradient Descent (SGD) and its Ruppert-Polyak averaged variant (ASGD) lie at the heart of modern large-scale learning, yet their theoretical properties in high-dimension…
Central Limit Theorems for Stochastic Gradient Descent Quantile Estimators
Ziyang Wei, Jiaqi Li, Likai Chen +1
This paper develops asymptotic theory for quantile estimation via stochastic gradient descent (SGD) with a constant learning rate. The quantile loss function is neither smooth nor…
Asymptotics of Stochastic Gradient Descent with Dropout Regularization in Linear Models
Jiaqi Li, Johannes Schmidt-Hieber, Wei Biao Wu
This paper proposes an asymptotic theory for online inference of the stochastic gradient descent (SGD) iterates with dropout regularization in linear regression. Specifically, we e…
Probabilistic Guarantees of Stochastic Recursive Gradient in Non-Convex Finite Sum Problems
Yanjie Zhong, Jiaqi Li, Soumendra Lahiri
This paper develops a new dimension-free Azuma-Hoeffding type bound on summation norm of a martingale difference sequence with random individual bounds. With this novel result, we…