Fast and Robust Online Inference with Stochastic Gradient Descent via Random Scaling

Abstract

We develop a new method of online inference for a vector of parameters estimated by the Polyak-Ruppert averaging procedure of stochastic gradient descent (SGD) algorithms. We leverage insights from time series regression in econometrics and construct asymptotically pivotal statistics via random scaling. Our approach is fully operational with online data and is rigorously underpinned by a functional central limit theorem. Our proposed inference method has a couple of key advantages over the existing methods. First, the test statistic is computed in an online fashion with only SGD iterates and the critical values can be obtained without any resampling methods, thereby allowing for efficient implementation suitable for massive online data. Second, there is no need to estimate the asymptotic variance and our inference method is shown to be robust to changes in the tuning parameters for SGD algorithms in simulation experiments with synthetic data.

arXiv comment29 pages, 8 figures, 8 tables

References in corpus (11)

• Adam: A Method for Stochastic Optimization
• Making Gradient Descent Optimal for Strongly Convex Stochastic Optimization
• Non-strongly-convex smooth stochastic approximation with convergence rate O(1/n)
• Statistical Inference for Model Parameters in Stochastic Gradient Descent
• HiGrad: Uncertainty Quantification for Online Learning and Stochastic Approximation
• On Linear Stochastic Approximation: Fine-grained Polyak-Ruppert and Non-Asymptotic Concentration
• Statistical Inference for the Population Landscape via Moment Adjusted Stochastic Gradients
• Normal Approximation for Stochastic Gradient Descent via Non-Asymptotic Rates of Martingale CLT
• Online Statistical Inference for Stochastic Optimization via Kiefer-Wolfowitz Methods
• Statistical Estimation and Inference via Local SGD in Federated Learning
• On Constructing Confidence Region for Model Parameters in Stochastic Gradient Descent via Batch Means

Cited by in corpus (3)

• Online Statistical Inference for Stochastic Optimization via Kiefer-Wolfowitz Methods
• When Composite Likelihood Meets Stochastic Approximation
• Statistical Estimation and Inference via Local SGD in Federated Learning