Nonparametric Heterogeneity Testing For Massive Data

Abstract

A massive dataset often consists of a growing number of (potentially) heterogeneous sub-populations. This paper is concerned about testing various forms of heterogeneity arising from massive data. In a general nonparametric framework, a set of testing procedures are designed to accommodate a growing number of sub-populations, denoted as $s$, with computational feasibility. In theory, their null limit distributions are derived as being nearly Chi-square with diverging degrees of freedom as long as $s$ does not grow too fast. Interestingly, we find that a lower bound on $s$ needs to be set for obtaining a sufficiently powerful testing result, so-called "blessing of aggregation." As a by-produc, a type of homogeneity testing is also proposed with a test statistic being aggregated over all sub-populations. Numerical results are presented to support our theory.

References in corpus (2)

• ANOVA for longitudinal data with missing values
• A Distributed One-Step Estimator

Cited by in corpus (6)

• Efficient Distributed Learning with Sparsity
• Positively Weighted Kernel Quadrature via Subsampling
• How Many Machines Can We Use in Parallel Computing for Kernel Ridge Regression?
• Additive Partially Linear Models for Massive Heterogeneous Data
• Distributed sequential method for analyzing massive data
• Oversampling Divide-and-conquer for Response-skewed Kernel Ridge Regression