Quantile Regression Under Memory Constraint

Abstract

This paper studies the inference problem in quantile regression (QR) for a large sample size $n$ but under a limited memory constraint, where the memory can only store a small batch of data of size $m$. A natural method is the naïve divide-and-conquer approach, which splits data into batches of size $m$, computes the local QR estimator for each batch, and then aggregates the estimators via averaging. However, this method only works when $n=o(m^2)$ and is computationally expensive. This paper proposes a computationally efficient method, which only requires an initial QR estimator on a small batch of data and then successively refines the estimator via multiple rounds of aggregations. Theoretically, as long as $n$ grows polynomially in $m$, we establish the asymptotic normality for the obtained estimator and show that our estimator with only a few rounds of aggregations achieves the same efficiency as the QR estimator computed on all the data. Moreover, our result allows the case that the dimensionality $p$ goes to infinity. The proposed method can also be applied to address the QR problem under distributed computing environment (e.g., in a large-scale sensor network) or for real-time streaming data.

References in corpus (3)

• Extremal Quantile Regression: An Overview
• Quantile Regression Under Memory Constraint
• Medians and Beyond: New Aggregation Techniques for Sensor Networks

Cited by in corpus (22)

• Quantile Regression Under Memory Constraint
• Distributed ARIMA Models for Ultra-long Time Series
• A review of distributed statistical inference
• First-order Newton-type Estimator for Distributed Estimation and Inference
• Distributed High-dimensional Regression Under a Quantile Loss Function
• Distributed Inference for Linear Support Vector Machine
• Using ARIMA to Predict the Expansion of Subscriber Data Consumption
• A Selective Review on Statistical Methods for Massive Data Computation: Distributed Computing, Subsampling, and Minibatch Techniques
• Distributed linear regression by averaging
• Federated Gaussian Process: Convergence, Automatic Personalization and Multi-fidelity Modeling
• Variance Reduced Median-of-Means Estimator for Byzantine-Robust Distributed Inference
• Fast and Robust Sparsity Learning over Networks: A Decentralized Surrogate Median Regression Approach
• Distributed Bootstrap for Simultaneous Inference Under High Dimensionality
• WONDER: Weighted one-shot distributed ridge regression in high dimensions
• Distributed Estimation for Principal Component Analysis: an Enlarged Eigenspace Analysis
• Distributed Estimation and Inference for Semi-parametric Binary Response Models
• Renewable Composite Quantile Method and Algorithm for Nonparametric Models with Streaming Data
• An Asymptotic Analysis of Minibatch-Based Momentum Methods for Linear Regression Models
• Distributed nonparametric regression imputation for missing response problems with large-scale data
• Divide-and-conquer methods for big data analysis
• Distributed Adaptive Huber Regression
• A Global Bias-Correction DC Method for Biased Estimation under Memory Constraint