Distributed Estimation and Inference with Statistical Guarantees
arXiv:1509.05457
Abstract
This paper studies hypothesis testing and parameter estimation in the context of the divide and conquer algorithm. In a unified likelihood based framework, we propose new test statistics and point estimators obtained by aggregating various statistics from subsamples of size , where is the sample size. In both low dimensional and high dimensional settings, we address the important question of how to choose as grows large, providing a theoretical upper bound on such that the information loss due to the divide and conquer algorithm is negligible. In other words, the resulting estimators have the same inferential efficiencies and estimation rates as a practically infeasible oracle with access to the full sample. Thorough numerical results are provided to back up the theory.
References in corpus (9)
- Nearly unbiased variable selection under minimax concave penalty
- Challenges of Big Data Analysis
- Confidence Intervals and Hypothesis Testing for High-Dimensional Regression
- Sure independence screening in generalized linear models with NP-dimensionality
- Divide and Conquer Kernel Ridge Regression: A Distributed Algorithm with Minimax Optimal Rates
- A Unified Framework for High-Dimensional Analysis of M-Estimators with Decomposable Regularizers
- Communication-efficient sparse regression: a one-shot approach
- A General Framework for Robust Testing and Confidence Regions in High-Dimensional Quantile Regression
- A Partially Linear Framework for Massive Heterogeneous Data
Cited by in corpus (34)
- Quantile Regression Under Memory Constraint
- Least Squares Approximation for a Distributed System
- First-order Newton-type Estimator for Distributed Estimation and Inference
- Efficient Distributed Learning with Sparsity
- Distributed Inference for Linear Support Vector Machine
- Distributed Estimation of Principal Eigenspaces
- A Distributed and Integrated Method of Moments for High-Dimensional Correlated Data Analysis
- Communication-efficient Distributed Sparse Linear Discriminant Analysis
- Nonparametric Heterogeneity Testing For Massive Data
- Debiased distributed learning for sparse partial linear models in high dimensions
- A Distributed One-Step Estimator
- Randomized maximum-contrast selection: subagging for large-scale regression
- Distributed Nonparametric Regression under Communication Constraints
- Simultaneous Inference for Massive Data: Distributed Bootstrap
- Fast Inference for Intractable Likelihood Problems using Variational Bayes
- Scalable and Efficient Statistical Inference with Estimating Functions in the MapReduce Paradigm for Big Data
- Distributed Bayesian Learning with Stochastic Natural-gradient Expectation Propagation and the Posterior Server
- Distributed Bootstrap for Simultaneous Inference Under High Dimensionality
- Distributed Estimation and Inference for Semi-parametric Binary Response Models
- Communication-efficient Distributed Estimation and Inference for Transelliptical Graphical Models
- Learning over inherently distributed data
- Distributed Learning of Finite Gaussian Mixtures
- Distributed estimation of principal support vector machines for sufficient dimension reduction
- Divide-and-conquer methods for big data analysis
- Removing the Curse of Superefficiency: an Effective Strategy For Distributed Computing in Isotonic Regression
- Distributed Community Detection for Large Scale Networks Using Stochastic Block Model
- Sample Splitting as an M-Estimator with Application to Physical Activity Scoring
- Meta Clustering for Collaborative Learning
- Statistical Validity and Consistency of Big Data Analytics: A General Framework
- Efficient Estimation for Generalized Linear Models on a Distributed System with Nonrandomly Distributed Data
- Divide and Conquer Local Average Regression
- Distributed Nearest Neighbor Classification
- Distributed sequential method for analyzing massive data
- : A Divide-and-conquer Algorithm for Large-scale Kernel Learning with Application to Clustering