Distributed Estimation and Inference with Statistical Guarantees

Abstract

This paper studies hypothesis testing and parameter estimation in the context of the divide and conquer algorithm. In a unified likelihood based framework, we propose new test statistics and point estimators obtained by aggregating various statistics from $k$ subsamples of size $n/k$, where $n$ is the sample size. In both low dimensional and high dimensional settings, we address the important question of how to choose $k$ as $n$ grows large, providing a theoretical upper bound on $k$ such that the information loss due to the divide and conquer algorithm is negligible. In other words, the resulting estimators have the same inferential efficiencies and estimation rates as a practically infeasible oracle with access to the full sample. Thorough numerical results are provided to back up the theory.

References in corpus (3)

• Nearly unbiased variable selection under minimax concave penalty
• Communication-efficient sparse regression: a one-shot approach
• A General Framework for Robust Testing and Confidence Regions in High-Dimensional Quantile Regression

Cited by in corpus (33)

• Quantile Regression Under Memory Constraint
• Least Squares Approximation for a Distributed System
• First-order Newton-type Estimator for Distributed Estimation and Inference
• Efficient Distributed Learning with Sparsity
• Distributed Inference for Linear Support Vector Machine
• Distributed Estimation of Principal Eigenspaces
• A Distributed and Integrated Method of Moments for High-Dimensional Correlated Data Analysis
• Communication-efficient Distributed Sparse Linear Discriminant Analysis
• Debiased distributed learning for sparse partial linear models in high dimensions
• Nonparametric Heterogeneity Testing For Massive Data
• A Distributed One-Step Estimator
• Simultaneous Inference for Massive Data: Distributed Bootstrap
• Fast Inference for Intractable Likelihood Problems using Variational Bayes
• Distributed Nonparametric Regression under Communication Constraints
• Distributed Bootstrap for Simultaneous Inference Under High Dimensionality
• Distributed Bayesian Learning with Stochastic Natural-gradient Expectation Propagation and the Posterior Server
• Scalable and Efficient Statistical Inference with Estimating Functions in the MapReduce Paradigm for Big Data
• Distributed Estimation and Inference for Semi-parametric Binary Response Models
• Learning over inherently distributed data
• Communication-efficient Distributed Estimation and Inference for Transelliptical Graphical Models
• Distributed Learning of Finite Gaussian Mixtures
• Distributed estimation of principal support vector machines for sufficient dimension reduction
• Divide-and-conquer methods for big data analysis
• Removing the Curse of Superefficiency: an Effective Strategy For Distributed Computing in Isotonic Regression
• Distributed Community Detection for Large Scale Networks Using Stochastic Block Model
• Efficient Estimation for Generalized Linear Models on a Distributed System with Nonrandomly Distributed Data
• Distributed Nearest Neighbor Classification
• Sample Splitting as an M-Estimator with Application to Physical Activity Scoring
• Statistical Validity and Consistency of Big Data Analytics: A General Framework
• Meta Clustering for Collaborative Learning
• Divide and Conquer Local Average Regression
• Distributed sequential method for analyzing massive data
• $D{C}^2$: A Divide-and-conquer Algorithm for Large-scale Kernel Learning with Application to Clustering