Statistical Inference for Model Parameters in Stochastic Gradient Descent

Abstract

The stochastic gradient descent (SGD) algorithm has been widely used in statistical estimation for large-scale data due to its computational and memory efficiency. While most existing works focus on the convergence of the objective function or the error of the obtained solution, we investigate the problem of statistical inference of true model parameters based on SGD when the population loss function is strongly convex and satisfies certain smoothness conditions. Our main contributions are two-fold. First, in the fixed dimension setup, we propose two consistent estimators of the asymptotic covariance of the average iterate from SGD: (1) a plug-in estimator, and (2) a batch-means estimator, which is computationally more efficient and only uses the iterates from SGD. Both proposed estimators allow us to construct asymptotically exact confidence intervals and hypothesis tests. Second, for high-dimensional linear regression, using a variant of the SGD algorithm, we construct a debiased estimator of each regression coefficient that is asymptotically normal. This gives a one-pass algorithm for computing both the sparse regression coefficients and confidence intervals, which is computationally attractive and applicable to online data.

73 pages

References in corpus (3)

• Confidence Intervals and Hypothesis Testing for High-Dimensional Regression
• Stochastic Optimization for Machine Learning
• Reconstruction from anisotropic random measurements

Cited by in corpus (12)

• A Simple Baseline for Bayesian Uncertainty in Deep Learning
• Generalization Bounds of SGLD for Non-convex Learning: Two Theoretical Viewpoints
• The Implicit Regularization of Stochastic Gradient Flow for Least Squares
• HiGrad: Uncertainty Quantification for Online Learning and Stochastic Approximation
• On Linear Stochastic Approximation: Fine-grained Polyak-Ruppert and Non-Asymptotic Concentration
• Fast and Robust Online Inference with Stochastic Gradient Descent via Random Scaling
• Statistical Inference for the Population Landscape via Moment Adjusted Stochastic Gradients
• Statistical Inference for Online Decision Making via Stochastic Gradient Descent
• On Scalable Inference with Stochastic Gradient Descent
• Normal Approximation for Stochastic Gradient Descent via Non-Asymptotic Rates of Martingale CLT
• On the Incorporation of Box-Constraints for Ensemble Kalman Inversion
• Approximate Newton-based statistical inference using only stochastic gradients