Nonparametric Heterogeneity Testing For Massive Data
arXiv:1601.06212
Abstract
A massive dataset often consists of a growing number of (potentially) heterogeneous sub-populations. This paper is concerned about testing various forms of heterogeneity arising from massive data. In a general nonparametric framework, a set of testing procedures are designed to accommodate a growing number of sub-populations, denoted as , with computational feasibility. In theory, their null limit distributions are derived as being nearly Chi-square with diverging degrees of freedom as long as does not grow too fast. Interestingly, we find that a lower bound on needs to be set for obtaining a sufficiently powerful testing result, so-called "blessing of aggregation." As a by-produc, a type of homogeneity testing is also proposed with a test statistic being aggregated over all sub-populations. Numerical results are presented to support our theory.
References in corpus (8)
- Concentration inequalities and asymptotic results for ratio type empirical processes
- ANOVA for longitudinal data with missing values
- Distributed Estimation and Inference with Statistical Guarantees
- Local and global asymptotic inference in smoothing spline models
- Parallelizing MCMC with Random Partition Trees
- Communication-efficient sparse regression: a one-shot approach
- Joint asymptotics for semi-nonparametric regression models with partially linear structure
- A Distributed One-Step Estimator
Cited by in corpus (6)
- Efficient Distributed Learning with Sparsity
- Positively Weighted Kernel Quadrature via Subsampling
- How Many Machines Can We Use in Parallel Computing for Kernel Ridge Regression?
- Additive Partially Linear Models for Massive Heterogeneous Data
- Distributed sequential method for analyzing massive data
- Oversampling Divide-and-conquer for Response-skewed Kernel Ridge Regression