PASS-GLM: polynomial approximate sufficient statistics for scalable Bayesian GLM inference

Abstract

Generalized linear models (GLMs) -- such as logistic regression, Poisson regression, and robust regression -- provide interpretable models for diverse data types. Probabilistic approaches, particularly Bayesian ones, allow coherent estimates of uncertainty, incorporation of prior information, and sharing of power across experiments via hierarchical models. In practice, however, the approximate Bayesian methods necessary for inference have either failed to scale to large data sets or failed to provide theoretical guarantees on the quality of inference. We propose a new approach based on constructing polynomial approximate sufficient statistics for GLMs (PASS-GLM). We demonstrate that our method admits a simple algorithm as well as trivial streaming and distributed extensions that do not compound error across computations. We provide theoretical guarantees on the quality of point (MAP) estimates, the approximate posterior, and posterior mean and uncertainty estimates. We validate our approach empirically in the case of logistic regression using a quadratic approximation and show competitive performance with stochastic gradient descent, MCMC, and the Laplace approximation in terms of speed and multiple measures of accuracy -- including on an advertising data set with 40 million data points and 20,000 covariates.

In Proceedings of the 31st Annual Conference on Neural Information Processing Systems (NIPS 2017). v3: corrected typos in Appendix A

References in corpus (7)

• Expectation Propagation for approximate Bayesian inference
• Coresets for Scalable Bayesian Logistic Regression
• Automatic Variational Inference in Stan
• Stochastic Bouncy Particle Sampler
• The Fundamental Incompatibility of Hamiltonian Monte Carlo and Data Subsampling
• Sequential Quantiles via Hermite Series Density Estimation
• Real-Time Machine Learning: The Missing Pieces

Cited by in corpus (6)

• Practical bounds on the error of Bayesian posterior approximations: A nonasymptotic approach
• Efficient non-conjugate Gaussian process factor models for spike count data using polynomial approximations
• Differentially Private Bayesian Inference for Generalized Linear Models
• No Free Lunch for Approximate MCMC
• Approximating posteriors with high-dimensional nuisance parameters via integrated rotated Gaussian approximation
• Scalable Gaussian Process Inference with Finite-data Mean and Variance Guarantees